Search

Patients with type 1 diabetes must continually decide how much insulin to inject before each meal to maintain blood glucose levels within a healthy range. Recent research has worked on a solution for this burden, showing the potential of reinforcement learning as an emerging approach for the task of controlling blood glucose levels. In this paper, we test and evaluate several deep Q-learning algorithms for automated and personalized blood glucose regulation in an in silico type 1 diabetes patient with the goal of estimating and delivering proper insulin doses. The proposed algorithms are model-free approaches with no prior information about the patient. We used the Hovorka model with meal variation and carbohydrate counting errors to simulate the patient included in this work. Our experiments compare different deep Q-learning extensions showing promising results controlling blood glucose levels, with some of the proposed algorithms outperforming standard baseline treatment. [ABSTRACT FROM AUTHOR]

Adaptive algorithms are widely used in deep learning because of their fast convergence. Among them, Adam is the most widely used algorithm. However, studies have shown that Adam's generalization ability is weak. AdaX is a variant of Adam, which introduces a novel second-order momentum, modifies the second-order moment of Adam, and has good generalization ability. However, these algorithms may fail to converge due to instability and extreme learning rates during training. In this paper, we propose a new adaptive and momental bound algorithm, called AdaXod, which characterizes of exponentially averaging the learning rate and is particularly useful for training deep neural networks. By setting an adaptively limited learning rate in the AdaX algorithm, the resultant AdaXod can effectively eliminate the problem of excessive learning rate in the later stage of neural networks training and thus results in stable training. We conduct extensive experiments on different datasets and verify the advantages of the AdaXod algorithm by comparing with other advanced adaptive optimization algorithms. AdaXod eliminates large learning rates during neural networks training and outperforms other optimizers, especially for some neural networks with complex structures, such as DenseNet. [ABSTRACT FROM AUTHOR]

Traditional path planning is mainly utilized for path planning in discrete action space, which results in incomplete ship navigation power propulsion strategies during the path search process. Moreover, reinforcement learning experiences low success rates due to its unbalanced sample collection and unreasonable design of reward function. In this paper, an environment framework is designed, which is constructed using the Box2D physics engine and employs a reward function, with the distance between the agent and arrival point as the main, and the potential field superimposed by boundary control, obstacles, and arrival point as the supplement. We also employ the state-of-the-art PPO (Proximal Policy Optimization) algorithm as a baseline for global path planning to address the issue of incomplete ship navigation power propulsion strategy. Additionally, a Beta policy-based distributed sample collection PPO algorithm is proposed to overcome the problem of unbalanced sample collection in path planning by dividing sub-regions to achieve distributed sample collection. The experimental results show the following: (1) The distributed sample collection training policy exhibits stronger robustness in the PPO algorithm; (2) The introduced Beta policy for action sampling results in a higher path planning success rate and reward accumulation than the Gaussian policy at the same training time; (3) When planning a path of the same length, the proposed Beta policy-based distributed sample collection PPO algorithm generates a smoother path than traditional path planning algorithms, such as A*, IDA*, and Dijkstra. [ABSTRACT FROM AUTHOR]

This paper proposes a data valuation algorithm for inertial measurement unit-based human activity recognition (IMU-based HAR) data based on meta reinforcement learning. Unlike previous studies that received feature-level input, the algorithm in this study added a feature extraction structure to the data valuation algorithm, and it can receive raw-level inputs and achieve excellent performance. As IMU-based HAR data are multivariate time-series data, the proposed algorithm incorporates an architecture capable of extracting both local and global features by inserting a transformer encoder after the one-dimensional convolutional neural network (1D-CNN) backbone in the data value estimator. In addition, the 1D-CNN-based stacking ensemble structure, which exhibits excellent efficiency and performance on IMU-based HAR data, is used as a predictor to supervise model training. The Berg balance scale (BBS) IMU-based HAR dataset and the public datasets, UCI-HAR, WISDM, and PAMAP2, are used for performance evaluation in this study. The valuation performance of the proposed algorithm is observed to be excellent on IMU-based HAR data. The rate of discovering corrupted data is higher than 96% on all datasets. In addition, classification performance is confirmed to be improved by the suppression of discovery of low-value data. [ABSTRACT FROM AUTHOR]

For vehicle-to-network communications, handover (HO) management enables vehicles to maintain the connection with the network while transiting through coverage areas of different base stations (BSs). However, the high mobility of vehicles means shorter connection periods with each BS that leads to frequent HOs, hence raises the necessity for optimal HO decision making for high quality infotainment services. Machine learning is capable of capturing underlying pattern via data driven methods to find optimal solutions to complex problems, and much learning-based HO optimization research has been conducted focusing on specific network setups. However, attention still needs to be paid to the actual deployment aspect and standardized datasets or simulation environments for evaluation. This paper proposes a deep reinforcement learning-based HO algorithm using the input parameters that are configurable in the existing measurement report of cellular networks. The performance of the proposed algorithm is evaluated using the well-known network simulator ns-3 with its official LTE module. A realistic network setup in the city center of Glasgow (U.K.) is configured with vehicle trajectories generated by the routes mobility model using Google Maps Directions API. Evaluation results reveal that the proposed algorithm significantly outperforms the A3 RSRP baseline with an average of 25.72% packet loss reduction per HO, suggesting significant improvement in quality of service of phone call and video streaming, etc. The proposed algorithm also has a small implementation cost compared to some state-of-the-art and should be deployed by a software update to a local BS controller. [ABSTRACT FROM AUTHOR]

In this paper, we present an iteration algorithm for the pricing of American options based on reinforcement learning. At each iteration, the method approximates the expected discounted payoff of stopping times and produces those closer to optimal. In the convergence analysis, a finite sample bound of the algorithm is derived. The algorithm is evaluated on a multi-dimensional Black-Scholes model and a symmetric stochastic volatility model, the numerical results implied that our algorithm is accurate and efficient for pricing high-dimensional American options. [ABSTRACT FROM AUTHOR]

This paper concerns the deployment problem of wireless sensor networks (WSNs) with mobile robotic sensor nodes for spatiotemporal monitoring. The proposed approach, deep reinforced learning tree (DRLT), utilizes deep reinforcement learning (DRL) to improve the efficiency of searching the most informative sampling locations. The parameterized sampling locations in an infinite horizon space are modeled according to their spatiotemporal correlations and subject to various constraints, including field estimation error and information gain. And the model-based information gain can be calculated efficiently over an infinite horizon. In this manner, the effectiveness of the sampling locations is learned through DRLT during the exploration by the robotic sensors. Then DRLT can instruct the robotic sensors to avoid unnecessary sampling locations in future iterations. Also, it is proved in this paper that the proposed algorithm is capable of searching for the near-optimal sampling locations effectively and guaranteeing a minimum field estimation error. Simulation based on national oceanic and atmospheric administration (NOAA) datasets is presented, which demonstrates the significant enhancements made by the proposed algorithm. Compared with the traditional approaches, such as the information theory-based greedy approach or random sampling, the proposed method shows a superior performance with regard to both estimation error and planning efficiency. [ABSTRACT FROM AUTHOR]

This review presents a high-level overview of the uses of machine learning (ML) to address several challenges in spatial auditory display research, primarily using head-related transfer functions. This survey also reviews and compares several categories of ML techniques and their application to virtual auditory reality research. This work addresses the use of ML techniques such as dimensionality reduction, unsupervised learning, supervised learning, reinforcement learning, and deep learning algorithms. The paper concludes with a discussion of the usage of ML algorithms to address specific spatial auditory display research challenges. [ABSTRACT FROM AUTHOR]

The increasing number of defunct and fragmented spacecraft poses a growing hazard to existing onorbit assets. The redundant continuum manipulator with high flexibility provides dual-arm robotic systems with apparent advantages in active debris removal missions in space. Existing autonomously-coordinated control approaches for dual-arm continuum manipulators require a real-time inverse kinematic solution and a security assurance mechanism for possible collisions, which are difficult to upscale for space debris capture systems with high-speed maneuverability. In this paper, we consider collision avoidance and input saturation control in proposing a multiagent reinforcement learning approach, named the multiagent twin delayed deep deterministic policy gradient (MATD3), to generate a real-time inverse kinematic solution for coordinated manipulators. During the training process, the MATD3 algorithm performs lower overestimation than the multiagent deep deterministic policy gradient (MADDPG) algorithm. Then, a feedback dynamics controller is designed for the continuum manipulators. Under the guidance of the policy networks, each agent can schedule the joint trajectory design online according to the collaborator and target debris information. During the capture operation, a competitive mechanism for the anticollision function is developed through reasonable reward functions to maintain dual arms at a safe distance. Simulation results show that the average accuracy of the proposed approach is 42% higher than that of MADDPG in inverse kinematic trajectory planning. The designed integrated tracking controller can effectively perform capture missions in the simulation environment. Multiagent reinforcement learning shows promise for future onorbit servicing missions. [ABSTRACT FROM AUTHOR]

The reinforcement learning algorithms based on policy gradient may fall into local optimal due to gradient disappearance during the update process, which in turn affects the exploration ability of the reinforcement learning agent. In order to solve the above problem, in this paper, the cross-entropy method (CEM) in evolution policy, maximum mean difference (MMD), and twin delayed deep deterministic policy gradient algorithm (TD3) are combined to propose a diversity evolutionary policy deep reinforcement learning (DEPRL) algorithm. By using the maximum mean discrepancy as a measure of the distance between different policies, some of the policies in the population maximize the distance between them and the previous generation of policies while maximizing the cumulative return during the gradient update. Furthermore, combining the cumulative returns and the distance between policies as the fitness of the population encourages more diversity in the offspring policies, which in turn can reduce the risk of falling into local optimal due to the disappearance of the gradient. The results in the MuJoCo test environment show that DEPRL has achieved excellent performance on continuous control tasks; especially in the Ant-v2 environment, the return of DEPRL ultimately achieved a nearly 20% improvement compared to TD3. [ABSTRACT FROM AUTHOR]

In recent years, unmanned aerial vehicles (UAVs) have unlocked numerous sensing applications, which are expected to add billions of dollars to the world economy in the next decade. To further improve the Quality-of-Service in these applications, the 3rd Generation Partnership Project has considered the use of terrestrial cellular networks to support UAV sensing services, also known as the cellular Internet of UAVs. In this paper, we consider a cellular Internet of UAVs, where the sensory data can be transmitted either to the base station via cellular links, or to the mobile devices by underlay UAV-to-Device (U2D) communications. To evaluate the freshness of the sensory data, the concept of age of information (AoI) is adopted, in which a lower AoI implies fresher data. Since UAVs’ AoIs are determined by their trajectories during sensing and transmission, we investigate the AoI minimization problem for UAVs by designing their trajectories. This problem is a Markov decision problem with an infinite state-action space, and thus we utilize multi-agent deep reinforcement learning to approximate the state-action space. Then, we propose a multi-UAV trajectory design algorithm to solve this problem. Simulation results show that our proposed algorithm can achieve a lower AoI than a greedy algorithm, policy gradient algorithm, and overlay U2D scheme. [ABSTRACT FROM AUTHOR]

The Internet of vehicles (IoV) is a large information interaction network that collects information on vehicles, roads and pedestrians. One of the important uses of vehicle networks is to meet the entertainment needs of driving users through communication between vehicles and roadside units (RSUs). Due to the limited storage space of RSUs, determining the content cached in each RSU is a key challenge. With the development of 5G and video editing technology, short video systems have become increasingly popular. Current widely used cache update methods, such as partial file precaching and content popularity- and user interest-based determination, are inefficient for such systems. To solve this problem, this paper proposes a QoE-driven edge caching method for the IoV based on deep reinforcement learning. First, a class-based user interest model is established. Compared with the traditional file popularity- and user interest distribution-based cache update methods, the proposed method is more suitable for systems with a large number of small files. Second, a quality of experience (QoE)-driven RSU cache model is established based on the proposed class-based user interest model. Third, a deep reinforcement learning method is designed to address the QoE-driven RSU cache update issue effectively. The experimental results verify the effectiveness of the proposed algorithm. [ABSTRACT FROM AUTHOR]

Deep reinforcement learning (DRL) is a promising technology to address the resource allocation problem for efficient data transmission in complex network environments. However, most DRL-based resource allocation algorithms suffer from limited feature extraction capabilities and lack scalability and generalization, especially in heterogeneous Industrial Internet of Things (IIoT) environments. In this paper, we develop a lifelong learning architecture that can integrate artificial intelligence (AI) algorithms into the heterogeneous IIoT network for efficient data transmission. Based on this, we propose an intelligent resource allocation algorithm based on dual-attention DRL (DADR) for forwarding node selection and channel access slot allocation in a specific network environment. The proposed DADR algorithm combines the advantages of multi-dimension convolutional attention and multi-head self-attention mechanisms. It can provide local- and global-feature fusion capabilities for distributed nodes while maximizing the performance of data transmission. Furthermore, we present a lifelong federated meta reinforcement learning (LFMRL) that can effectively utilize prior knowledge and enable the DRL agent quickly adapt to a new environment. Specifically, LFMRL adopts a federated meta learning-based knowledge fusion algorithm to fuse the knowledge of learned DADR algorithms and iteratively update the shared model, thereby improving the scalability and generalization of the shared model in heterogeneous IIoT environments. In addition, a simple and efficient knowledge transfer mechanism is enabled to accelerate the DRL model convergence by transferring the knowledge of the shared model to the new environment. Simulation results demonstrate the effectiveness of the proposed algorithms in terms of energy efficiency, data transmission reliability, and network stability. Compared to DADR and FedAvg algorithms, LFMRL algorithm can further reduce the energy consumption, training time, and average forwarding node switching times, while improving packet delivery rate to 99.2%. • A novel lifelong learning architecture for resource allocation in heterogeneous IIoT. • Dual-attention deep reinforcement learning provides local- and global- feature fusion. • Federated meta learning based knowledge fusion learns the commonness of models. • Reinforcement learning based knowledge transfer learns the properties of model. • Knowledge fusion and knowledge transfer provide lifelong learning ability for system. [ABSTRACT FROM AUTHOR]

In this paper, we study the handover (HO), and power allocation problem in a two-tier heterogeneous network (HetNet), which consists of a macro base station, and some millimeter-wave (mmWave) small base stations. We establish an HO management, and power allocation scheme to maximize the overall throughput while reducing the HO frequency. In particular, considering the interrelationship among decisions made by different user equipments (UEs), we first model the HO, and power allocation problem as a fully cooperative multi-agent task, in which all agents, i.e., UEs, have the same target. Then, to solve the multi-agent task, and get decentralized policies for each UE, we develop a multi-agent reinforcement learning (MARL) algorithm based on the proximal policy optimization (PPO) method, by introducing the centralized training with decentralized execution framework. That is, we use global information to train policies for each UE, and after the training is finished, each UE obtains a decentralized policy, which can be implemented only based on each UE's local observation. Specially, we introduce the counterfactual baseline to address the credit assignment problem in centralized learning. Due to the centralized training, the decentralized polices learned by multi-agent PPO (MAPPO) can work more cooperatively. Finally, the simulation results demonstrate that our method can achieve better performance comparing with other existing works. [ABSTRACT FROM AUTHOR]

With respect to the problem of the low accuracy of traditional building energy prediction methods, this paper proposes a novel prediction method for building energy consumption, which is based on the seamless integration of the deep neural network and transfer reinforcement learning (DNN-TRL). The method introduces a stack denoising autoencoder to extract the deep features of the building energy consumption, and shares the hidden layer structure to transfer the common information between different building energy consumption problems. The output of the DNN model is used as the input of the Sarsa algorithm to improve the prediction performance of the target building energy consumption. To verify the performance of the DNN-TRL algorithm, based on the data recorded by American Power Balti Gas and Electric Power Company, and compared with Sarsa, ADE-BPNN, and BP-Adaboost algorithms, the experimental results show that the DNN-TRL algorithm can effectively improve the prediction accuracy of the building energy consumption. [ABSTRACT FROM AUTHOR]

The vehicular edge computing is considered an enabling technology for intelligent and connected vehicles since the optimization of communication and computing on edge has a significant impact on driving safety and efficiency. In this paper, with the road traffic assignment to “proactively” reshape the spatiotemporal distribution of resource demands, we investigate the joint service migration and mobility optimization problem for vehicular edge computing. The goal is to meet the service delay requirements of vehicular edge computing with minimum migration cost and travel time. As service migration and mobility optimization are coupled, the joint scheduling problem suffers from the curse of dimensionality, which cannot be solved in real time by centralized algorithms. To this end, a multi-agent deep reinforcement learning (MADRL) algorithm is proposed to maximize the composite utility of communication, computing, and route planning in a distributed way. In the MADRL algorithm, a two-branch convolution based deep Q-network is constructed to coordinate migration action and routing action. Extensive experimental results show that the proposed algorithm is scalable and substantially reduces service delay, migration cost and travel time as compared with the existing baselines. [ABSTRACT FROM AUTHOR]

As urban traffic condition is diverse and complicated, applying reinforcement learning to reduce traffic congestion becomes one of the hot and promising topics. Especially, how to coordinate the traffic light controllers of multiple intersections is a key challenge for multi-agent reinforcement learning (MARL). Most existing MARL studies are based on traditional $Q$ -learning, but unstable environment leads to poor learning in the complicated and dynamic traffic scenarios. In this paper, we propose a novel multi-agent recurrent deep deterministic policy gradient (MARDDPG) algorithm based on deep deterministic policy gradient (DDPG) algorithm for traffic light control (TLC) in vehiclar networks. Specifically, the centralized learning in each critic network enables each agent to estimate the policies of other agents in the decision-making process and each agent can coordinate with each other, alleviating the problem of poor learning performance caused by environmental instability. The decentralized execution enables each agent to make decisions independently. We share parameters in actor networks to speed up the training process and reduce the memory footprint. The addition of LSTM is beneficial to alleviate the instability of the environment caused by partial observable state. We utilize surveillance cameras and vehicular networks to collect status information for each intersection. Unlike previous work, we have not only considered the vehicle but also considered the pedestrians waiting to pass through the intersection. Moreover, we also set different priorities for buses and ordinary vehicles. The experimental results in a vehicular network show that our method can run stably in various scenarios and coordinate multiple intersections, which significantly reduces vehicle congestion and pedestrian congestion. [ABSTRACT FROM AUTHOR]

Wireless virtual reality (VR) is predicted to become a killer application in 5G and beyond, which provides an immersive experience and revolutionizes the way people communicate. It is well-known that rendering is the key performance bottleneck in wireless VR systems, especially for VR games. However, real-time rendering and data correlation are ignored by most researchers. In this paper, we propose an adaptive VR framework that enables high-quality wireless VR in future mmWave-enabled wireless networks with mobile edge computing (MEC), where real-time VR rendering tasks can be offloaded to MEC servers adaptively and the caching capability of MEC servers enables further performance improvement. First, we formulate the addressed problem to maximize the quality of experience (QoE) of the users, where association, adaptive offloading mode selection, and caching policy are jointly optimized. Considering the high complexity of the addressed problem, we then propose a distributed learning approach consisting of an offline training phase and an online running phase, which maintains scalability and adaptation capability. The offline phase is based on deep reinforcement learning (DRL) while the latter utilizes game theory. At last, simulation results show the superiority of the proposed algorithm over the other baseline algorithms in terms of QoE utility values, latency, and convergence time. [ABSTRACT FROM AUTHOR]

The chemical mechanical polishing (CMP) process usually suffers from drift and shift in the Run-to-Run material removal process due to the wear and replacement of the polishing pad, lacking of in-suit measurements of the product quality of interest and other environment variations. This paper proposed a deep reinforcement learning (DRL)-based run-to-run (R2R) controller for the CMP process. Firstly, deep reinforcement learning is effectively utilized as a training algorithm of the R2R controller, which is a model-free controller to take a decision with infinite horizon information and thus improves the control performance; Secondly, a novel policy network is embeded to the DRL model, which divides the network into linear and nonlinear part explicitly to improve the prediction performance of the R2R controller on process changes. Finally, a special reward function is proposed to improve the training of the R2R controller, which trades off between target tracing and fluctuations of production parameters. The effectiveness of the proposed controller is validated on a CMP process. The testing results illustrate that the DRL-based R2R controller can precisely trace the desired target of material removal rate (MRR) and is very effective to control various process variations online. [ABSTRACT FROM AUTHOR]

In the current unmanned aircraft systems (UASs) for sensing services, unmanned aerial vehicles (UAVs) transmit their sensory data to terrestrial mobile devices over the unlicensed spectrum. However, the interference from surrounding terminals is uncontrollable due to the opportunistic channel access. In this paper, we consider a cellular Internet of UAVs to guarantee the Quality-of-Service (QoS), where the sensory data can be transmitted to the mobile devices either by UAV-to-Device (U2D) communications over cellular networks, or directly through the base station (BS). Since UAVs’ sensing and transmission may influence their trajectories, we study the trajectory design problem for UAVs in consideration of their sensing and transmission. This is a Markov decision problem (MDP) with a large state-action space, and thus, we utilize multi-agent deep reinforcement learning (DRL) to approximate the state-action space, and then propose a multi-UAV trajectory design algorithm to solve this problem. Simulation results show that our proposed algorithm can achieve a higher total utility than policy gradient algorithm and single-agent algorithm. [ABSTRACT FROM AUTHOR]

The fifth generation of wireless communications (5G) promises massive increases in traffic volume and data rates, as well as improved reliability in voice calls. Jointly optimizing beamforming, power control, and interference coordination in a 5G wireless network to enhance the communication performance to end users poses a significant challenge. In this paper, we formulate the joint design of beamforming, power control, and interference coordination as a non-convex optimization problem to maximize the signal to interference plus noise ratio (SINR) and solve this problem using deep reinforcement learning. By using the greedy nature of deep Q-learning to estimate future rewards of actions and using the reported coordinates of the users served by the network, we propose an algorithm for voice bearers and data bearers in sub-6 GHz and millimeter wave (mmWave) frequency bands, respectively. The algorithm improves the performance measured by SINR and sum-rate capacity. In realistic cellular environments, the simulation results show that our algorithm outperforms the link adaptation industry standards for sub-6 GHz voice bearers. For data bearers in the mmWave frequency band, our algorithm approaches the maximum sum rate capacity, but with less than 4% of the required run time. [ABSTRACT FROM AUTHOR]

Proposed by Maddah-Ali and Niesen, a coded caching scheme has been verified to alleviate the load of networks efficiently. Recently, a new technique called placement delivery array (PDA) was proposed to characterize the coded caching scheme. In this paper, we consider a caching system in the scope of ultra dense networks (UDNs). Each base station (BS) has a finite cache and stores some contents. We propose an efficient coded content caching scheme called double coded caching to make the transmission robust to in-and-out wireless network quality. Then the dynamic caching and multicast scheduling are considered to jointly minimize the average delay and power of the content-centric wireless networks. This stochastic optimization problem can be formulated as a Markov decision process (MDP) with unknown transition probabilities and large state space. We propose a deep reinforcement learning approach to deal with the decision problem. Our algorithm uses a variational auto-encoder (VAE) neural network to approximate the state sufficiently, and uses a weighted double Q-learning scheme to reduce variance and overestimation of the Q function. Numerical results demonstrate that the proposed double coded caching scheme increases the probability of the successful transmission, and the caching and scheduling policy can effectively reduce the delay and the power consumption. [ABSTRACT FROM AUTHOR]

Deep deterministic policy gradient algorithm operating over continuous space of actions has attracted great attention for reinforcement learning. However, the exploration strategy through dynamic programming within the Bayesian belief state space is rather inefficient even for simple systems. Another problem is the sequential and iterative training data with autonomous vehicles subject to the law of causality, which is against the i.i.d. (independent identically distributed) data assumption of the training samples. This usually results in failure of the standard bootstrap when learning an optimal policy. In this paper, we propose a framework of m-out-of-n bootstrapped and aggregated multiple deep deterministic policy gradient to accelerate the training process and increase the performance. Experiment results on the 2D robot arm game show that the reward gained by the aggregated policy is 10%–50% better than those gained by subpolicies. Experiment results on the open racing car simulator (TORCS) demonstrate that the new algorithm can learn successful control policies with less training time by 56.7%. Analysis on convergence is also given from the perspective of probability and statistics. These results verify that the proposed method outperforms the existing algorithms in both efficiency and performance. [ABSTRACT FROM AUTHOR]

Unmanned Aerial Vehicles (UAVs) with Microwave Power Transfer (MPT) capability provide a practical means to deploy a large number of wireless powered sensing devices into areas with no access to persistent power supplies. The UAV can charge the sensing devices remotely and harvest their data. A key challenge is online MPT and data collection in the presence of on-board control of a UAV (e.g., patrolling velocity) for preventing battery drainage and data queue overflow of the devices, while up-to-date knowledge on battery level and data queue of the devices is not available at the UAV. In this paper, an on-board deep Q-network is developed to minimize the overall data packet loss of the sensing devices, by optimally deciding the device to be charged and interrogated for data collection, and the instantaneous patrolling velocity of the UAV. Specifically, we formulate a Markov Decision Process (MDP) with the states of battery level and data queue length of devices, channel conditions, and waypoints given the trajectory of the UAV; and solve it optimally with Q-learning. Furthermore, we propose the on-board deep Q-network that enlarges the state space of the MDP, and a deep reinforcement learning based scheduling algorithm that asymptotically derives the optimal solution online, even when the UAV has only outdated knowledge on the MDP states. Numerical results demonstrate that our deep reinforcement learning algorithm reduces the packet loss by at least 69.2%, as compared to existing non-learning greedy algorithms. [ABSTRACT FROM AUTHOR]

Deep Reinforcement Learning systems are now a hot topic in Machine Learning for their effectiveness in many complex tasks, but their application in safety-critical domains (e.g., robot control or self-autonomous driving) remains dangerous without mechanism to detect and prevent risk situations. In Deep RL, such risk is mostly in the form of adversarial attacks, which introduce small perturbations to sensor inputs with the aim of changing the network-based decisions and thus cause catastrophic situations. In the light of these dangers, a promising line of research is that of providing these Deep RL algorithms with suitable defenses, especially when deploying in real environments. This paper suggests that this line of research could be greatly improved by the concepts from the existing research field of Safe Reinforcement Learning, which has been postulated as a family of RL algorithms capable of providing defenses against many forms of risks. However, the connections between Safe RL and the design of defenses against adversarial attacks in Deep RL remain largely unexplored. This paper seeks to explore precisely some of these connections. In particular, this paper proposes to reuse some of the concepts from existing Safe RL algorithms to create a novel and effective instance-based defense for the deployment stage of Deep RL policies. The proposed algorithm uses a risk function based on how far a state is from the state space known by the agent, that allows identifying and preventing adversarial situations. The success of the proposed defense has been evaluated in 4 Atari games. [ABSTRACT FROM AUTHOR]

Planetary soft landing has been studied extensively due to its promising application prospects. In this paper, a soft landing control algorithm based on deep reinforcement learning (DRL) with good convergence property is proposed. First, the soft landing problem of the powered descent phase is formulated and the theoretical basis of Reinforcement Learning (RL) used in this paper is introduced. Second, to make it easier to converge, a reward function is designed to include process rewards like velocity tracking reward, solving the problem of sparse reward. Then, by including the fuel consumption penalty and constraints violation penalty, the lander can learn to achieve velocity tracking goal while saving fuel and keeping attitude angle within safe ranges. Then, simulations of training are carried out under the frameworks of Deep deterministic policy gradient (DDPG), Twin Delayed DDPG (TD3), and Soft Actor Critic (SAC), respectively, which are of the classical RL frameworks, and all converged. Finally, the trained policy is deployed into velocity tracking and soft landing experiments, results of which demonstrate the validity of the algorithm proposed. [ABSTRACT FROM AUTHOR]

This paper addresses the challenge of minimizing training time for the control of Heating, Ventilation, and Air-conditioning (HVAC) systems with online Reinforcement Learning (RL). This is done by developing a novel approach to Multi-Agent Reinforcement Learning (MARL) to HVAC systems. In this paper, the environment formed by the HVAC system is formulated as a Markov Game (MG) in a general sum setting. The MARL algorithm is designed in a decentralized structure, where only relevant states are shared between agents, and actions are shared in a sequence, which are sensible from a system's point of view. The simulation environment is a domestic house located in Denmark and designed to resemble an average house. The heat source in the house is an air-to-water heat pump, and the HVAC system is an Underfloor Heating system (UFH). The house is subjected to weather changes from a data set collected in Copenhagen in 2006, spanning the entire year except for June, July, and August, where heat is not required. It is shown that: (1) When comparing Single Agent Reinforcement Learning (SARL) and MARL, training time can be reduced by 70% for a four temperature-zone UFH system, (2) the agent can learn and generalize over seasons, (3) the cost of heating can be reduced by 19% or the equivalent to 750 kWh of electric energy per year for an average Danish domestic house compared to a traditional control method, and (4) oscillations in the room temperature can be reduced by 40% when comparing the RL control methods with a traditional control method. [ABSTRACT FROM AUTHOR]

A pursuit–evasion game is a classical maneuver confrontation problem in the multi-agent systems (MASs) domain. An online decision technique based on deep reinforcement learning (DRL) was developed in this paper to address the problem of environment sensing and decision-making in pursuit–evasion games. A control-oriented framework developed from the DRL-based multi-agent deep deterministic policy gradient (MADDPG) algorithm was built to implement multi-agent cooperative decision-making to overcome the limitation of the tedious state variables required for the traditionally complicated modeling process. To address the effects of errors between a model and a real scenario, this paper introduces adversarial disturbances. It also proposes a novel adversarial attack trick and adversarial learning MADDPG (A2-MADDPG) algorithm. By introducing an adversarial attack trick for the agents themselves, uncertainties of the real world are modeled, thereby optimizing robust training. During the training process, adversarial learning was incorporated into our algorithm to preprocess the actions of multiple agents, which enabled them to properly respond to uncertain dynamic changes in MASs. Experimental results verified that the proposed approach provides superior performance and effectiveness for pursuers and evaders, and both can learn the corresponding confrontational strategy during training. [ABSTRACT FROM AUTHOR]

In an autonomous vehicle, the lane following algorithm is an important component, which is a basic function of autonomous driving. However, the existing lane following system has a few shortcomings: first, the control method it adopts requires an accurate system model, and different vehicles have different parameters, which needs a lot of parameter calibration work. The second is that it may fail on road sections where the lateral acceleration requirements of vehicles are large, such as large curves. Third, its decision-making system is defined based on rules, which has disadvantages: it is difficult to formulate; human subjective factors cannot guarantee objectivity; coverage is difficult to guarantee. In recent years, the deep deterministic policy gradient (DDPG) algorithm has been widely used in the field of autonomous driving due to its strong nonlinear fitting ability and generalization performance. However, the DDPG algorithm has overestimated state action values and large cumulative errors, low training efficiency and other issues. Therefore, this paper improves the DDPG algorithm based on the double critic networks and priority experience replay mechanism. Then this paper proposes a lane following method based on this algorithm. Experiment shows that the algorithm can achieve excellent following results under various road conditions. [ABSTRACT FROM AUTHOR]

The demand for bandwidth-intensive and delay-sensitive services is surging daily with the development of 5G technology, resulting in fierce competition for scarce radio resources. Power domain Nonorthogonal Multiple Access (NOMA) technologies can dramatically improve system capacity and spectrum efficiency. Unlike existing NOMA scheduling that mainly focuses on fairness, this paper proposes a power control solution for uplink hybrid OMA and PD-NOMA in dual dynamic environments: dynamic and imperfect channel information together with the random user-specific hierarchical quality of service (QoS). This paper models the power control problem as a nonconvex stochastic, which aims to maximize system energy efficiency while guaranteeing hierarchical user QoS requirements. Then, the problem is formulated as a partially observable Markov decision process (POMDP). Owing to the difficulty of modeling time-varying scenes, the urgency of fast convergency, the adaptability in a dynamic environment, and the continuity of the variables, a Deep Reinforcement Learning (DRL)-based method is proposed. This paper also transforms the hierarchical QoS constraint under the NOMA serial interference cancellation (SIC) scene to fit DRL. The simulation results verify the effectiveness and robustness of the proposed algorithm under a dual uncertain environment. As compared with the baseline Particle Swarm Optimization algorithm (PSO), the proposed DRL-based method has demonstrated satisfying performance. [ABSTRACT FROM AUTHOR]

This paper proposes an optimal route and charging station selection (RCS) algorithm based on model-free deep reinforcement learning (DRL) to overcome the uncertainty issues of the traffic conditions and dynamic arrival charging requests. The proposed DRL based RCS algorithm aims to minimize the total travel time of electric vehicles (EV) charging requests from origin to destination using the selection of the optimal route and charging station considering dynamically changing traffic conditions and unknown future requests. In this paper, we formulate this RCS problem as a Markov decision process model with unknown transition probability. A Deep Q network has been adopted with function approximation to find the optimal electric vehicle charging station (EVCS) selection policy. To obtain the feature states for each EVCS, we define the traffic preprocess module, charging preprocess module and feature extract module. The proposed DRL based RCS algorithm is compared with conventional strategies such as minimum distance, minimum travel time, and minimum waiting time. The performance is evaluated in terms of travel time, waiting time, charging time, driving time, and distance under the various distributions and number of EV charging requests. [ABSTRACT FROM AUTHOR]

Behind-the-meter distributed energy resources (DERs), including building solar photovoltaic (PV) technology and electric battery storage, are increasingly being considered as solutions to support carbon reduction goals and increase grid reliability and resiliency. However, dynamic control of these resources in concert with traditional building loads, to effect efficiency and demand flexibility, is not yet commonplace in commercial control products. Traditional rule-based control algorithms do not offer integrated closed-loop control to optimize across systems, and most often, PV and battery systems are operated for energy arbitrage and demand charge management, and not for the provision of grid services. More advanced control approaches, such as MPC control have not been widely adopted in industry because they require significant expertise to develop and deploy. Recent advances in deep reinforcement learning (DRL) offer a promising option to optimize the operation of DER systems and building loads with reduced setup effort. However, there are limited studies that evaluate the efficacy of these methods to control multiple building subsystems simultaneously. Additionally, most of the research has been conducted in simulated environments as opposed to real buildings. This paper proposes a DRL approach that uses a deep deterministic policy gradient algorithm for integrated control of HVAC and electric battery storage systems in the presence of on-site PV generation. The DRL algorithm, trained on synthetic data, was deployed in a physical test building and evaluated against a baseline that uses the current best-in-class rule-based control strategies. Performance in delivering energy efficiency, load shift, and load shed was tested using price-based signals. The results showed that the DRL-based controller can produce cost savings of up to 39.6% as compared to the baseline controller, while maintaining similar thermal comfort in the building. The project team has also integrated the simulation components developed during this work as an OpenAIGym environment and made it publicly available so that prospective DRL researchers can leverage this environment to evaluate alternate DRL algorithms. • Traditional controls do not integrate distributed energy resources (DER) systems. • Deterministic policy gradient algorithm is proposed to optimize the operation of DER. • The algorithm is deployed and evaluated in a physical test building. • Tests include energy efficiency, load shift, and load shed. • An OpenAI Gym environment is made it publicly available for other researchers. [ABSTRACT FROM AUTHOR]

• The combination of mobile devices and edge computing reduces the total computing overhead and makes resource allocation reasonable. • Evaluate system performance by the weighted sum of delay and energy consumption. • Change the parameter update method to accelerate the convergence speed of the deep reinforcement learning algorithm. • A neural network is used to approximate the behavior value function, and a target q network is used to update the target. With the arrival of the 5th generation mobile networks (5 G) era, the data needed by mobile devices (MDs) is explosively growing. High-consumption, low-latency applications are huge challenges for resource-constrained Internet of things (IoT) devices. Mobile edge computing overcomes the limitations of computing resources on MDs by offloading tasks generated by MDs and assigning them to nearby MEC servers. Therefore, mobile edge computing (MEC) becomes important. This paper presents a task offloading strategy for the multi-device multi-server system. To meet the task requirements of different MDs, we formulate an overhead minimization problem to optimize the delay and energy consumption of the system. We propose the Double Deep Q Network (Double-DQN) algorithm to perform location selection strategies for tasks generated on the mobile devices and allocate respective computing resources. Simulation results show that the algorithm can allocate resources reasonably and reduce the overhead of the entire system. [Display omitted] [ABSTRACT FROM AUTHOR]

Wireless networking using GHz or THz spectra has encouraged mobile service providers to deploy small cells to improve link quality and cell capacity using mmWave backhaul links. As green networking for less CO2 emission is mandatory to confront global climate change, we need energy efficient network management for such denser small-cell heterogeneous networks (HetNets) that already suffer from observable power consumption. We establish a dual-objective optimization model that minimizes energy consumption by switching off unused small cells while maximizing user throughput, which is a mixed integer linear problem (MILP). Recently, the deep reinforcement learning (DRL) algorithm has been applied to many NP-hard problems of the wireless networking field, such as radio resource allocation, association and power saving, which can induce a near-optimal solution with fast inference time as an online solution. In this paper, we investigate the feasibility of the DRL algorithm for a dual-objective problem, energy efficient routing and throughput maximization, which has not been explored before. We propose a proximal policy (PPO)-based multi-objective algorithm using the actor-critic model that is realized as an optimistic linear support framework in which the PPO algorithm searches for feasible solutions iteratively. Experimental results show that our algorithm can achieve throughput and energy savings comparable to the CPLEX. [ABSTRACT FROM AUTHOR]

In the task of interactive image segmentation, the Inside-Outside Guidance (IOG) algorithm has demonstrated superior segmentation performance leveraging Inside-Outside Guidance information. Nevertheless, we observe that the inconsistent input between training and testing when selecting the inside point will result in significant performance degradation. In this paper, a deep reinforcement learning framework, named Inside Point Localization Network (IPL-Net), is proposed to infer the suitable position for the inside point to help the IOG algorithm. Concretely, when a user first clicks two outside points at the symmetrical corner locations of the target object, our proposed system automatically generates the sequence of movement to localize the inside point. We then perform the IOG interactive segmentation method for precisely segmenting the target object of interest. The inside point localization problem is difficult to define as a supervised learning framework because it is expensive to collect image and their corresponding inside points. Therefore, we formulate this problem as Markov Decision Process (MDP) and then optimize it with Dueling Double Deep Q-Network (D3QN). We train our network on the PASCAL dataset and demonstrate that the network achieves excellent performance. [ABSTRACT FROM AUTHOR]

The Deep Reinforcement Learning (DRL) algorithm, Proximal Policy Optimization (PPO2), is deployed on a custom spacecraft (S/C) build and loss model to determine if an Artificial Intelligence (AI) can learn to monitor satellite constellation health and determine an optimal replacement strategy. A custom environment is created to simulate how S/C are built, launched, generate revenue, and finally decay. The reinforcement learning agent successfully learned an optimal policy for two models: a Simplified Model where the financial cost of actions is ignored; and an Advanced Model where the financial cost of actions is a major element. In both models the AI monitors the constellations and takes multiple strategic and tactical actions to replace satellites to maintain constellation performance. The Simplified Model showed that the PPO2 algorithm was able to converge on an optimal solution after ∼ 200,000 simulations. The Advanced Model was much more difficult for the AI to learn, and thus, the performance drops during the early episodes, but eventually converges to an optimal policy at ∼ 25,000,000 simulations. With the Advanced Model, the AI is taking actions that are successfully providing strategies for constellation management and satellite replacements which include these actions' financial implications. Thus, the methods in this paper provide initial research developments towards a real-world tool and an AI application that can aid various Aerospace businesses in managing Low Earth Orbit (LEO) constellations. This type of AI application may become imperative for deploying and maintaining small satellite mega-constellations. [ABSTRACT FROM AUTHOR]

Earth observation satellite task scheduling research plays a key role in space-based remote sensing services. An effective task scheduling strategy can maximize the utilization of satellite resources and obtain larger objective observation profits. In this paper, inspired by the success of deep reinforcement learning in optimization domains, the deep deterministic policy gradient algorithm is adopted to solve a time-continuous satellite task scheduling problem. Moreover, an improved graph-based minimum clique partition algorithm is proposed for preprocessing in the task clustering phase by considering the maximum task priority and the minimum observation slewing angle under constraint conditions. Experimental simulation results demonstrate that the deep reinforcement learning-based task scheduling method is feasible and performs much better than traditional metaheuristic optimization algorithms, especially in large-scale problems. [ABSTRACT FROM AUTHOR]

Because of the high penetration of renewable energies and the installation of new control devices, modern distribution networks are faced with voltage regulation challenges. Recently, the rapid development of artificial intelligence technology has introduced new solutions for optimal control problems with high dimensions and dynamics. In this paper, a deep reinforcement learning method is proposed to solve the two-timescale optimal voltage control problem. All control variables are assigned to different agents, and discrete variables are solved by a deep Q network (DQN) agent while the continuous variables are solved by a deep deterministic policy gradient (DDPG) agent. All agents are trained simultaneously with specially designed reward aiming at minimizing long-term average voltage deviation. Case study is executed on a modified IEEE-123 bus system, and the results demonstrate that the proposed algorithm has similar or even better performance than the model-based optimal control scheme and has high computational efficiency and competitive potential for online application. [ABSTRACT FROM AUTHOR]

Public websites offer information on a variety of topics and services and are accessed by users with varying skills to browse the kind of electronic document repositories. However, the complex website structure and diversity of web browsing behavior create a challenging task for click prediction. This paper presents the results of a novel reinforcement learning approach to model user browsing patterns in a hierarchically ordered municipal website. We study how accurate predictor the browsing history is, when the target pages are not immediate next pages pointed by hyperlinks, but appear a number of levels down the hierarchy. We compare traditional type of baseline classifiers' performance against our reinforcement learning-based training algorithm. [ABSTRACT FROM AUTHOR]

The research on autonomous driving based on deep reinforcement learning algorithms is a research hotspot. Traditional autonomous driving requires human involvement, and the autonomous driving algorithms based on supervised learning must be trained in advance using human experience. To deal with autonomous driving problems, this paper proposes an improved end-to-end deep deterministic policy gradient (DDPG) algorithm based on the convolutional block attention mechanism, and it is called multi-input attention prioritized deep deterministic policy gradient algorithm (MAPDDPG). Both the actor network and the critic network of the model have the same structure with symmetry. Meanwhile, the attention mechanism is introduced to help the vehicles focus on useful environmental information. The experiments are conducted in the open racing car simulator (TORCS)and the results of five experiment runs on the test tracks are averaged to obtain the final result. Compared with the state-of-the-art algorithm, the maximum reward increases from 62,207 to 116,347, and the average speed increases from 135 km/h to 193 km/h, while the number of success episodes to complete a circle increases from 96 to 147. Also, the variance of the distance from the vehicle to the center of the road is compared, and the result indicates that the variance of the DDPG is 0.6 m while that of the MAPDDPG is only 0.2 m. The above results indicate that the proposed MAPDDPG achieves excellent performance. [ABSTRACT FROM AUTHOR]

How to operate an unmanned aerial vehicle (UAV) safely and efficiently in an interactive environment is challenging. A large amount of research has been devoted to improve the intelligence of a UAV while performing a mission, where finding an optimal maneuver decision-making policy of the UAV has become one of the key issues when we attempt to enable the UAV autonomy. In this paper, we propose a maneuver decision-making algorithm based on deep reinforcement learning, which generates efficient maneuvers for a UAV agent to execute the airdrop mission autonomously in an interactive environment. Particularly, the training set of the learning algorithm by the Prioritized Experience Replay is constructed, that can accelerate the convergence speed of decision network training in the algorithm. It is shown that a desirable and effective maneuver decision-making policy can be found by extensive experimental results. [ABSTRACT FROM AUTHOR]

• Online network reconfiguration is introduced to manage distribution networks. • A Deep Q-learning algorithm is used to find the optimal network topology. • The proposed algorithm is more scalable than other algorithms. • The computation time of the algorithm is low enough for practical applications. • The algorithm can increase DRES utilization without heavy capital investment. With increasing number of distributed renewable energy sources integrated in power distribution networks, network security issues such as line overloading or bus voltage violations are becoming increasingly common. Traditional capital-intensive system reinforcements could lead to overinvestment. Moreover, active network management solutions, which have emerged as important alternatives, may become a financial burden for distribution system operators or reduce profits for owners of distributed renewable energy sources, or both. To address these limitations, this paper proposes an online network reconfiguration scheme based on a deep reinforcement learning approach. In this scheme, the distribution network operator modifies the network topology to change the power flow when the reliability of network is threatened. Because the variability of distributed renewable energy is large in self-sufficient distribution networks, the reconfiguration process needs to be performed online within short time intervals, which involves the use of conventional algorithms. To solve this problem efficiently, a deep q-learning model is utilized to determine the optimal network topology. Performances of proposed and other algorithms were compared in modified CIGRE 14-bus and IEEE 123-bus test network, as well as varying penalties for frequent switching operation in consideration of physical characteristic of the network. Simulation results demonstrated that the proposed algorithm showed almost identical performances with brute-force search algorithm in both test networks, satisfying network constraints over almost all timespans. Further, the proposed method required very small computation times - under a second per each state and its scalability was verified by comparing the computation time between two test networks. [ABSTRACT FROM AUTHOR]

• Genetic-State Grouping Algorithm guarantees enhancing the performance of RL agents. • The genetic algorithm has successfully been combined with Monte Carlo Tree Search. • Video game provides a valid proving ground for testing AI's capability. Although Reinforcement learning has already been considered one of the most important and well-known techniques of machine learning, its applicability remains limited in the real-world problems due to its long initial learning time and unstable learning. Especially, the problem of an overwhelming number of the branching factors under real-time constraint still stays unconquered, demanding a new method for the next generation of reinforcement learning. In this paper, we propose Genetic State-Grouping Algorithm based on deep reinforcement learning. The core idea is to divide the entire set of states into a few state groups. Each group consists of states that are mutually similar, thus representing their common features. The state groups are then processed with the Genetic Optimizer, which finds outstanding actions. These steps help the Deep Q Network avoid excessive exploration, thereby contributing to the significant reduction of initial learning time. The experiment on the real-time fighting video game (FightingICE) shows the effectiveness of our proposed approach. [ABSTRACT FROM AUTHOR]

• Apply a state-of-the-art deep reinforcement learning algorithm to the PM problem. • Formulate the PM problem as an MDP with guidance of domain knowledge. • Use a data-driven modeling method to build a fast simulator for learning. • The DRL agent learns to perform group maintenance and opportunistic maintenance. • Proposed method outperforms age-dependent policy and opportunistic policy. In the manufacturing industry, the preventive maintenance (PM) is a common practice to reduce random machine failures by replacing/repairing the aged machines or parts. The decision on when and where the preventive maintenance needs to be carried out is nontrivial due to the complex and stochastic nature of a serial production line with intermediate buffers. In order to improve the cost efficiency of the serial production lines, a deep reinforcement learning based approach is proposed to obtain PM policy. A novel modeling method for the serial production line is adopted during the learning process. A reward function is proposed based on the system production loss evaluation. The algorithm based on the Double Deep Q-Network is applied to learn the PM policy. Using the simulation study, the learning algorithm is proved effective in delivering PM policy that leads to an increased throughput and reduced cost. Interestingly, the learned policy is found to frequently conduct "group maintenance" and "opportunistic maintenance", although their concepts and rules are not provided during the learning process. This finding further demonstrates that the problem formulation, the proposed algorithm and the reward function setting in this paper are effective. [ABSTRACT FROM AUTHOR]

In this paper, a model-free deep reinforcement learning (DRL) strategy is presented with an artificial neural network (ANN) as reaction simulation environment, to obtain a fed-batch control strategy for an experimental bioreactor. The proposed method is a fundamental attempt to control reactions by employing state-of-the-art machine learning tools without the aid of well-established mechanistic understanding of the reaction system. This application utilizes the Asynchronous Advantage Actor-Critic (A3C) algorithm, a member of the DRL family, that takes advantage of actor-critic algorithm and asynchronous learning by parallel learning agents to achieve stability and efficiency of the learning process. The resulting controller demonstrates robust performance in the fed-batch bioreactor since it can be adjusted to meet varying constraining factors including nutrient limitations and culture lengths. Results are presented for a bioreactor that produces cyanobacterial-phycocyanin (C-PC) in Plectonema sp. UTEX 1541. Experimental validations show a 52.1% increase in the product yield, and a 20.1% increase in C-PC concentration compared to a control group with the same total nutrient input replenished in a non-optimized manner. [ABSTRACT FROM AUTHOR]

With advances in smart grid technologies, demand response has played a major role in improving the reliability of grids and reduce the cost for customers. Implementing the demand response scheme for industry is more necessary than for other sectors, because its energy consumption is often considered the largest. This paper proposes a multi-agent deep reinforcement learning based demand response scheme for energy management of discrete manufacturing systems. In this regard, the industrial manufacturing system is initially formulated as a partially-observable Markov game; then, a multi-agent deep deterministic policy gradient algorithm is adopted to obtain the optimal schedule for different machines. A typical lithium-ion battery assembly manufacturing system is used to demonstrate the effectiveness of the proposed scheme. Simulation results show that the presented demand response algorithm can minimize electricity costs and maintain production tasks, as compared to a benchmark without demand response. Moreover, the performance of the multi-agent deep reinforcement learning approach against a mathematical model method is investigated. [ABSTRACT FROM AUTHOR]

Since path planning for multi-arm manipulators is a complicated high-dimensional problem, effective and fast path generation is not easy for the arbitrarily given start and goal locations of the end effector. Especially, when it comes to deep reinforcement learning-based path planning, high-dimensionality makes it difficult for existing reinforcement learning-based methods to have efficient exploration which is crucial for successful training. The recently proposed soft actor–critic (SAC) is well known to have good exploration ability due to the use of the entropy term in the objective function. Motivated by this, in this paper, a SAC-based path planning algorithm is proposed. The hindsight experience replay (HER) is also employed for sample efficiency and configuration space augmentation is used in order to deal with complicated configuration space of the multi-arms. To show the effectiveness of the proposed algorithm, both simulation and experiment results are given. By comparing with existing results, it is demonstrated that the proposed method outperforms the existing results. [ABSTRACT FROM AUTHOR]