Search

The Internet of vehicles (IoV) is a large information interaction network that collects information on vehicles, roads and pedestrians. One of the important uses of vehicle networks is to meet the entertainment needs of driving users through communication between vehicles and roadside units (RSUs). Due to the limited storage space of RSUs, determining the content cached in each RSU is a key challenge. With the development of 5G and video editing technology, short video systems have become increasingly popular. Current widely used cache update methods, such as partial file precaching and content popularity- and user interest-based determination, are inefficient for such systems. To solve this problem, this paper proposes a QoE-driven edge caching method for the IoV based on deep reinforcement learning. First, a class-based user interest model is established. Compared with the traditional file popularity- and user interest distribution-based cache update methods, the proposed method is more suitable for systems with a large number of small files. Second, a quality of experience (QoE)-driven RSU cache model is established based on the proposed class-based user interest model. Third, a deep reinforcement learning method is designed to address the QoE-driven RSU cache update issue effectively. The experimental results verify the effectiveness of the proposed algorithm. [ABSTRACT FROM AUTHOR]

In the order picking process of the warehouse center, considering the rapid increase in the volume of orders arriving at the picking center at the time of the promotional festival, a hybrid operation mode with multiple picking tables is used to meet the picking requirements of the huge number of real-time orders. Therefore, in this paper, a hybrid picking mode is proposed, taking into account both the idle degree of picking stations and their order item centers of gravity, and a new reinforcement learning algorithm embedding mechanism (PRL) with placeholder control is designed to solve the problem of a huge number of real-time item orders arriving at the picking center system on promotional holidays and in inconsistent quantities, and numerical simulations are performed for this algorithm. The experimental results show that the PRL algorithm in hybrid picking mode can handle a huge number of orders simultaneously and improve picking efficiency effectively. [ABSTRACT FROM AUTHOR]

Path planning of intelligent driving vehicles in emergencies is a hot research issue, this paper proposes a new method of the best path selection for the intelligent driving vehicles to solve this problem. Based on the prior knowledge applied reinforcement learning strategy and the searching- optimized A* algorithm, we designed a hybrid algorithm to help intelligent driving vehicles selecting the best path in the traffic network in emergencies including limited height, width, weight, accident, and traffic jam. Through simulation experiments and scene experiments, it is proved that the proposed algorithm has good stability, high efficiency, and practicability. [ABSTRACT FROM AUTHOR]

Based on the clinical states of the patient, dynamic treatment regime technology can provide various therapeutic methods, which is helpful for medical treatment policymaking. Reinforcement learning is an important approach for developing this technology. In order to implement the reinforcement learning algorithm efficiently, the computation of health data is usually outsourced to the untrustworthy cloud server. However, it may leak, falsify, or delete private health data. Encryption is a common method for solving this problem. But the cloud server is difficult to calculate encrypted health data. In this paper, based on Cheon et al.'s approximate homomorphic encryption scheme, we first propose secure computation protocols for implementing comparison, maximum, exponentiation, and division. Next, we design a homomorphic reciprocal of square root protocol firstly, which only needs one approximate computation. Based on the proposed secure computation protocols, we design a secure asynchronous advantage actor-critic reinforcement learning algorithm for the first time. Then, it is used to implement a secure treatment decision-making algorithm. Simulation results show that our secure computation protocols and algorithms are feasible. [ABSTRACT FROM AUTHOR]

Penetration testing is an effective way to test and evaluate cybersecurity by simulating a cyberattack. However, the traditional methods deeply rely on domain expert knowledge, which requires prohibitive labor and time costs. Autonomous penetration testing is a more efficient and intelligent way to solve this problem. In this paper, we model penetration testing as a Markov decision process problem and use reinforcement learning technology for autonomous penetration testing in large scale networks. We propose an improved deep Q-network (DQN) named NDSPI-DQN to address the sparse reward problem and large action space problem in large-scale scenarios. First, we reasonably integrate five extensions to DQN, including noisy nets, soft Q-learning, dueling architectures, prioritized experience replay, and intrinsic curiosity model to improve the exploration efficiency. Second, we decouple the action and split the estimators of the neural network to calculate two elements of action separately, so as to decrease the action space. Finally, the performance of algorithms is investigated in a range of scenarios. The experiment results demonstrate that our methods have better convergence and scaling performance. [ABSTRACT FROM AUTHOR]

The reinforcement learning algorithms based on policy gradient may fall into local optimal due to gradient disappearance during the update process, which in turn affects the exploration ability of the reinforcement learning agent. In order to solve the above problem, in this paper, the cross-entropy method (CEM) in evolution policy, maximum mean difference (MMD), and twin delayed deep deterministic policy gradient algorithm (TD3) are combined to propose a diversity evolutionary policy deep reinforcement learning (DEPRL) algorithm. By using the maximum mean discrepancy as a measure of the distance between different policies, some of the policies in the population maximize the distance between them and the previous generation of policies while maximizing the cumulative return during the gradient update. Furthermore, combining the cumulative returns and the distance between policies as the fitness of the population encourages more diversity in the offspring policies, which in turn can reduce the risk of falling into local optimal due to the disappearance of the gradient. The results in the MuJoCo test environment show that DEPRL has achieved excellent performance on continuous control tasks; especially in the Ant-v2 environment, the return of DEPRL ultimately achieved a nearly 20% improvement compared to TD3. [ABSTRACT FROM AUTHOR]

In reinforcement learning (RL), the exploration/exploitation (E/E) dilemma is a very crucial issue, which can be described as searching between the exploration of the environment to find more profitable actions, and the exploitation of the best empirical actions for the current state. We focus on the single trajectory RL problem where an agent is interacting with a partially unknown MDP over single trajectories, and try to deal with the E/E in this setting. Given the reward function, we try to find a good E/E strategy to address the MDPs under some MDP distribution. This is achieved by selecting the best strategy in mean over a potential MDP distribution from a large set of candidate strategies, which is done by exploiting single trajectories drawn from plenty of MDPs. In this paper, we mainly make the following contributions: (1) We discuss the strategy-selector algorithm based on formula set and polynomial function. (2) We provide the theoretical and experimental regret analysis of the learned strategy under an given MDP distribution. (3) We compare these methods with the 'state-of-the-art' Bayesian RL method experimentally. [ABSTRACT FROM AUTHOR]

In recent years, unmanned aerial vehicles (UAVs) have unlocked numerous sensing applications, which are expected to add billions of dollars to the world economy in the next decade. To further improve the Quality-of-Service in these applications, the 3rd Generation Partnership Project has considered the use of terrestrial cellular networks to support UAV sensing services, also known as the cellular Internet of UAVs. In this paper, we consider a cellular Internet of UAVs, where the sensory data can be transmitted either to the base station via cellular links, or to the mobile devices by underlay UAV-to-Device (U2D) communications. To evaluate the freshness of the sensory data, the concept of age of information (AoI) is adopted, in which a lower AoI implies fresher data. Since UAVs’ AoIs are determined by their trajectories during sensing and transmission, we investigate the AoI minimization problem for UAVs by designing their trajectories. This problem is a Markov decision problem with an infinite state-action space, and thus we utilize multi-agent deep reinforcement learning to approximate the state-action space. Then, we propose a multi-UAV trajectory design algorithm to solve this problem. Simulation results show that our proposed algorithm can achieve a lower AoI than a greedy algorithm, policy gradient algorithm, and overlay U2D scheme. [ABSTRACT FROM AUTHOR]

This paper proposes a DRL-based cache content update policy in the cache-enabled network to improve the cache hit ratio and reduce the average latency. In contrast to the existing policies, a more practical cache scenario is considered in this work, in which the content requests vary by both time and location. Considering the constraint of the limited cache capacity, the dynamic content update problem is modeled as a Markov decision process (MDP). Besides that, the deep Q-learning network (DQN) algorithm is utilised to solve the MDP problem. Specifically, the neural network is optimised to approximate the Q value where the training data are chosen from the experience replay memory. The DQN agent derives the optimal policy for the cache decision. Compared with the existing policies, the simulation results show that our proposed policy is 56%–64% improved in terms of the cache hit ratio and 56%–59% decreased in terms of the average latency. [ABSTRACT FROM AUTHOR]

This study proposes a novel adaptive distributed recurrent SAC (Soft Actor–Critic) control method to address the leader–follower formation control problem of omnidirectional mobile robots. Our method successfully eliminates the reliance on the complete state of the leader and achieves the task of formation solely using the pose between robots. Moreover, we develop a novel recurrent SAC reinforcement learning framework that ensures that the controller exhibits good transient and steady-state characteristics to achieve outstanding control performance. We also present an episode-based memory replay buffer and sampling approaches, along with a unique normalized reward function, which expedites the recurrent SAC reinforcement learning formation framework to converge rapidly and receive consistent incentives across various leader–follower tasks. This facilitates better learning and adaptation to the formation task requirements in different scenarios. Furthermore, to bolster the generalization capability of our method, we normalized the state space, effectively eliminating differences between formation tasks of different shapes. Different shapes of leader–follower formation experiments in the Gazebo simulator achieve excellent results, validating the efficacy of our method. Comparative experiments with traditional PID and common network controllers demonstrate that our method achieves faster convergence and greater robustness. These simulation results provide strong support for our study and demonstrate the potential and reliability of our method in solving real-world problems. [ABSTRACT FROM AUTHOR]

This paper characterizes a class of repeated moral hazard problem where the agent's replacement action is necessary for assuring the performance of important materials and equipment. However, due to the principal's costly monitoring, the continuously decreasing performance caused by the agent's successive shirking behaviors will significantly harm the principal's profits or welfare. For sake of performance assurance and monitoring efficiency, this paper adopts two reinforcement learning algorithms to find and test the strategies on the final path satisfying the conditions of (Restricted) Experience Based Equilibrium. Experimental results in a case of quality assurance of grain demonstrate the effectiveness of the model and the solution methods. Practically, this framework especially fits a kind of government-enterprise relation, such as in the field of emergency materials management and safety production management. Given the increasing need of developing systems for the government's management, this paper aims to provide applicable strategies and guidelines from a theoretical perspective. [ABSTRACT FROM AUTHOR]

Recently, studies on early-exit mechanisms have emerged to reduce the computational cost during the inference process of deep learning models. However, most existing early-exit architectures simply determine early exiting based only on a target confidence level in the prediction, without any consideration of the computational cost. Such an early-exit criterion fails to balance accuracy and cost, making it difficult to use in various environments. To address this problem, we propose a novel, cost-effective early-exit architecture in which an early-exit criterion is designed based on the Markov decision process (MDP). Since the early-exit decisions within an early-exit model are sequential, we model them as an MDP problem to maximize accuracy as much as possible while minimizing the computational cost. Then, we develop a cost-effective early-exit algorithm using reinforcement learning that solves the MDP problem. For each input sample, the algorithm dynamically makes early-exit decisions considering the relative importance of accuracy and computational cost in a given environment, thereby balancing the trade-off between accuracy and cost regardless of the environment. Consequently, it can be used in various environments, even in a resource-constrained environment. Through extensive experiments, we demonstrate that our proposed architecture can effectively balance the trade-off in different environments, while the existing architectures fail to do so since they focus only on reducing their cost while preventing the degradation of accuracy. [ABSTRACT FROM AUTHOR]

Planetary soft landing has been studied extensively due to its promising application prospects. In this paper, a soft landing control algorithm based on deep reinforcement learning (DRL) with good convergence property is proposed. First, the soft landing problem of the powered descent phase is formulated and the theoretical basis of Reinforcement Learning (RL) used in this paper is introduced. Second, to make it easier to converge, a reward function is designed to include process rewards like velocity tracking reward, solving the problem of sparse reward. Then, by including the fuel consumption penalty and constraints violation penalty, the lander can learn to achieve velocity tracking goal while saving fuel and keeping attitude angle within safe ranges. Then, simulations of training are carried out under the frameworks of Deep deterministic policy gradient (DDPG), Twin Delayed DDPG (TD3), and Soft Actor Critic (SAC), respectively, which are of the classical RL frameworks, and all converged. Finally, the trained policy is deployed into velocity tracking and soft landing experiments, results of which demonstrate the validity of the algorithm proposed. [ABSTRACT FROM AUTHOR]

The bilateral spot electricity market is very complicated because all generation units and demands must strategically bid in this market. Considering renewable resource penetration, the high variability and the non-dispatchable nature of these intermittent resources make it more difficult to model and simulate the dynamic bidding process and the equilibrium in the bilateral spot electricity market, which makes developing fast and reliable market modeling approaches a matter of urgency nowadays. In this paper, a Gradient Descent Continuous Actor-Critic algorithm is proposed for hour-ahead bilateral electricity market modeling in the presence of renewable resources because this algorithm can solve electricity market modeling problems with continuous state and action spaces without causing the "curse of dimensionality" and has low time complexity. In our simulation, the proposed approach is implemented on an IEEE 30-bus test system. The adequate performance of our proposed approach--such as reaching Nash Equilibrium results after enough iterations of training are tested and verified, and some conclusions about the relationship between increasing the renewable power output and participants' bidding strategy, locational marginal prices, and social welfare--is also evaluated. Moreover, the comparison of our proposed approach with the fuzzy Q-learning-based electricity market approach implemented in this paper confirms the superiority of our proposed approach in terms of participants' profits, social welfare, average locational marginal prices, etc. [ABSTRACT FROM AUTHOR]

At present, there is a serious disconnect between online teaching and offline teaching in English MOOC large-scale hybrid teaching recommendation platform, which is mainly due to the problems of cold start and matrix sparsity in the recommendation algorithm, and it is difficult to fully tap the user's interest characteristics because it only considers the user's rating but neglects the user's personalized evaluation. In order to solve the above problems, this paper proposes to use reinforcement learning thought and user evaluation factors to realize the online and offline hybrid English teaching recommendation platform. First, the idea of value function estimation in reinforcement learning is introduced, and the difference between user state value functions is used to replace the previous similarity calculation method, thus alleviating the matrix sparsity problem. The learning rate is used to control the convergence speed of the weight vector in the user state value function to alleviate the cold start problem. Second, by adding the learning of the user evaluation vector to the value function estimation of the state value function, the state value function of the user can be estimated approximately and the discrimination degree of the target user can be reflected. Experimental results show that the proposed recommendation algorithm can effectively alleviate the cold start and matrix sparsity problems existing in the current collaborative filtering recommendation algorithm and can dig deep into the characteristics of users' interests and further improve the accuracy of scoring prediction. [ABSTRACT FROM AUTHOR]

Multi-agent reinforcement learning (MARL) has been successfully applied in many fields. In MARL, the policy evaluation problem is one of crucial problems. In order to solve this problem, distributed Temporal-Difference (TD) learning algorithm is one of the most popular methods in a cooperative manner. Despite its empirical success, however, the theory of the adaptive variant of distributed TD learning still remain limited. To fill this gap, we propose an adaptive distributed temporal-difference algorithm (referred to as MS - ADTD) under Markovian sampling over time-varying networks. Furthermore, we rigorously analyze the convergence of MS - ADTD , the theoretical results show that the local estimation can converge linearly to the optimal neighborhood. Meanwhile, the theoretical results are verified by simulation experiments. • This paper proposes a distributed adaptive TD learning algorithm under Markovian sampling. • It theoretically analyzes the non-asymptotic convergence performance of the proposed algorithm. • It verifies the performance of the proposed algorithm by experiments. [ABSTRACT FROM AUTHOR]

Guiding an aircraft to 4D waypoints at a certain heading is a multi-dimensional goal aircraft guidance problem. In order to improve the performance and solve this problem, this paper proposes a multi-layer RL approach. The approach enables the autopilot in an ATC simulator to guide an aircraft to 4D waypoints at certain latitude, longitude, altitude, heading, and arrival time, respectively. To be specific, a multi-layer RL approach is proposed to simplify the neural network structure and reduce the state dimensions. A shaped reward function that involves the potential function and Dubins path method is applied. Experimental and simulation results show that the proposed approach can significantly improve the convergence efficiency and trajectory performance. Furthermore, the results indicate possible application prospects in team aircraft guidance tasks, since the aircraft can directly approach a goal without waiting in a specific pattern, thereby overcoming the problem of current ATC simulators. [ABSTRACT FROM AUTHOR]

In this paper, the fully cooperative multi-agent system is studied, in which all of the agents share the same common goal. The main difficulty in such systems is the coordination problem: how to ensure that the individual decisions of the agents lead to jointly optimal decisions for the group? Firstly, a multi-agent reinforcement learning algorithm combining traditional Q-learning with observation-based teammate modeling techniques, called TM_Qlearning, is presented and evaluated. Several new cooperative action selection strategies are then suggested to improve the multi-agent coordination and accelerate learning, especially in the case of unknown and temporary dynamic environments. The effectiveness of combining TM_Qlearning with the new proposals is demonstrated using the hunting game. [ABSTRACT FROM AUTHOR]

In this paper, we propose an adaptive obstacle avoidance algorithm based on DDPG (Deep Deterministic Policy Gradient) and DWA (Dynamic-Window Approach) to study the obstacle avoidance problem of robots in complex continuous state space. First, the obstacle avoidance problem is converted into an optimal learning incentive problem, and the self-learning of the obstacle avoidance policy is realized based on DDPG; second, the DWA obstacle avoidance trajectory evaluation function is optimized using the DDPG reward incentive mechanism, and the Experience Replay mechanism; finally, the algorithm model is simulated. The experiments show that the model can significantly circumvent the deficiency of the DWA algorithm in limiting to the optimal local solution in a complex environment and solve the action output problem in the continuous velocity and turning angle value interval of the robot; through the trial and error interaction with the environment and the trajectory evaluation incentive feedback, the obstacle avoidance passing ability of the robot in a complex environment is improved. [Display omitted] • We propose an adaptive obstacle avoidance algorithm based on DDPG (Deep Deterministic Policy Gradient) and DWA (Dynamic Window Approach), named the DDPG-DWA. • The DDPG-DWA algorithm can solve the problem of action output in the continuous velocity and turning angle value range of the robot. • The DDPG-DWA algorithm can avoid the problem of DWA algorithm easily being limited to local obstacle avoidance optimal solutions in complex environments. • The model is suitable for path obstacle avoidance of intelligent agents in complex, narrow and irregular environments. [ABSTRACT FROM AUTHOR]

Active object recognition (AOR) provides a paradigm where an agent can capture additional evidence by purposefully changing its viewpoint to improve the quality of recognition. One of themost concerned problems in AOR is viewpoint planning (VP) which refers to developing a policy to determine the next viewpoints of the agent. A research trend is to solve the VP problem with reinforcement learning, namely to use the viewpoint transitions explored by the agent to train the VP policy. However, most research discards the trained transitions, whichmay lead to an inefficient use of the explored transitions. To solve this challenge, we present a novel VP method with transition management based on reinforcement learning, which can reuse the explored viewpoint transitions. To be specific, a learning framework of the VP policy is first established via the deterministic policy gradient theory, which provides an opportunity to reuse the explored transitions. Then, we design a scheme of viewpoint transition management that can store the explored transitions and decide which transitions are used for the policy learning. Finally, within the framework, we develop an algorithm based on twin delayed deep deterministic policy gradient and the designed scheme to train the VP policy. Experiments on the public and challenging dataset GERMS show the effectiveness of our method in comparison with several competing approaches. [ABSTRACT FROM AUTHOR]

Product information is constantly changing in actual production; examples include batch due date changes, batch size changes, and the arrival of emergency batches. However, most studies of seru production have focused on the static seru systems. Therefore, this study investigated the total tardiness of the dynamic seru production considering seru formation changes. When product information changes, four decision processes were considered for optimal solutions: 1) selection of seru(s) for re-formation (SSR); 2) batch scheduling on the seru(s) waiting for re-formation (BSSWR); 3) seru re-formation; and 4) seru re-scheduling. A dynamic seru production model was established here based on the above process, and a phased intelligent algorithm was proposed for solving this problem. SSR was optimized using a genetic algorithm. The earliest due date (EDD) heuristic algorithm was used for BSSWR. Moreover, an adaptive cooperative coevolution algorithm was proposed, in which Q-learning was used for determining whether to optimize seru re-formation or seru re-scheduling. After extensive experiments, the dynamic seru production system considering the seru formation changes reduced the total tardiness by an average of 54% compared with the original solution, and by an average of 15% compared with the dynamic seru production system considering only the seru scheduling changes. [ABSTRACT FROM AUTHOR]

Reformulating the history matching problem from a least-square mathematical optimization problem into a Markov Decision Process introduces a method in which reinforcement learning can be utilized to solve the problem. This method provides a mechanism where an artificial deep neural network agent can interact with the reservoir simulator and find multiple different solutions to the problem. Such a formulation allows for solving the problem in parallel by launching multiple concurrent environments enabling the agent to learn simultaneously from all the environments at once, achieving significant speed up. [ABSTRACT FROM AUTHOR]

Reinforcement-learning (RL) algorithms have made great achievements in many scenarios. However, in large-scale traffic signal control (TSC) scenarios, RL still falls into local optima when controlling multiple signal lights. To solve this problem, we propose a novel goal-based multi-agent hierarchical model (GMHM). Specifically, we divide the traffic environment into several regions. The region contains a virtual manager and several workers who control the traffic lights. The manager assigns goals to each worker by observing the environment, and the worker makes decisions according to the environment state and the goal. For the worker, we adapted the goal-based multi-agent deep deterministic policy gradient (MADDPG) algorithm combined with hierarchical reinforcement learning. In this way, we simplify tasks and allow agents to cooperate more efficiently. We carried out experiments on both grid traffic scenarios and real-world scenarios in the SUMO simulator. The experimental results show the performance advantages of our algorithm compared with state-of-the-art algorithms. [ABSTRACT FROM AUTHOR]

Machine learning research has been able to solve problems in multiple domains. Machine learning represents an open area of research for solving optimisation problems. The optimisation problems can be solved using a metaheuristic algorithm, which can find a solution in a reasonable amount of time. However, the time required to find an appropriate metaheuristic algorithm, that would have the convenient configurations to solve a set of optimisation problems properly presents a problem. The proposal described in this article contemplates an approach that automatically creates metaheuristic algorithms given a set of optimisation problems. These metaheuristic algorithms are created by modifying their logical structure via the execution of an evolutionary process. This process employs an extension of the reinforcement learning approach that considers multi-agents in their environment, and a learning agent composed of an analysis process and a process of modification of the algorithms. The approach succeeded in creating a metaheuristic algorithm that managed to solve different continuous domain optimisation problems from the experiments performed. The implications of this work are immediate because they describe a basis for the generation of metaheuristic algorithms in an online-evolution. [ABSTRACT FROM AUTHOR]

Students generally attempt to improve themselves with different manners in their spare time. Inspired by this fact, this paper presents a multi-objective elitist feedback teaching–learning-based optimization (MEFTO) algorithm for multi-objective optimization problems. To promote the exploration capacity and convergence of the basic teaching–learning-based optimization (TLBO), a feedback phase that simulates the spare-time learning phenomenon is introduced at the end of the learner phase. Students should compare their score with the average class score to select an appropriate means for further improvement. Poorly performing students can learn from the teacher directly for rapid improvement, whereas high-performing students prefer to motivate themselves for reinforcement learning. Non-dominated sorting is incorporated to permit this heuristic to solve problems with several objectives. Crowding distance calculation is adopted to maintain the diversity of the obtained solution set in a single run. Elitism strategy is also employed to provide a great improvement for the algorithm. The performance of the MEFTO algorithm is compared with seven other well-known algorithms by using a set of unconstrained benchmark test problems and three constrained engineering optimization problems. The qualitative and quantitative results indicate that the proposed algorithm can provide considerably competitive results and outperforms the other algorithms. • A novel multi-objective evolutionary algorithm (MEFTO) is developed. • The developed algorithm is evaluated on a series of benchmark functions. • The performance of the MEFTO is demonstrated using various metrics. • The developed algorithm is applied to solve three constrained engineering problem. [ABSTRACT FROM AUTHOR]

Abstract: Q-Learning is based on value iteration and remains the most popular choice for solving Markov Decision Problems (MDPs) via reinforcement learning (RL), where the goal is to bypass the transition probabilities of the MDP. Approximate policy iteration (API) is another RL technique, not as widely used as Q-Learning, based on modified policy iteration. In this paper, we present and analyze an API algorithm for discounted reward based on (i) a classical temporal differences update for policy evaluation and (ii) simulation-based mean estimation for policy improvement. Further, we analyze for convergence API algorithms based on Q-factors for (i) discounted reward and (ii) for average reward MDPs. The average reward algorithm is based on relative value iteration; we also present results from some numerical experiments with it. [Copyright &y& Elsevier]

This paper investigates the problem of impact-time-control and proposes a learning-based computational guidance algorithm to solve this problem. The proposed guidance algorithm is developed based on a general prediction-correction concept: the exact time-to-go under proportional navigation guidance with realistic aerodynamic characteristics is estimated by a deep neural network and a biased command to nullify the impact time error is developed by utilizing the emerging reinforcement learning techniques. To deal with the problem of insufficient training data, a transfer-ensemble learning approach is proposed to train the deep neural network. The deep neural network is augmented into the reinforcement learning block to resolve the issue of sparse reward that has been observed in typical reinforcement learning formulation. Extensive numerical simulations are conducted to support the proposed algorithm. [ABSTRACT FROM AUTHOR]

• We are the first to address the NP-hard Budgeted Maximum Coverage Problem. • We propose an algorithm that combines reinforcement learning with local search. • We introduce the first set of benchmark instances for the problem. • The proposed algorithm outperforms the comparison algorithms and the CPLEX solver. The Budgeted Maximum Coverage Problem (BMCP) is a general model with a number of real-world applications. Given n elements with nonnegative profits, m subsets of elements with nonnegative weights and a total budget, the BMCP aims to select some subsets such that the total weight of the selected subsets does not exceed the budget, while the total profit of the associated elements is maximized. BMCP is NP-hard and thus computationally challenging. We investigate for the first time an effective practical algorithm for solving this problem, which combines reinforcement learning and local search. The algorithm iterates through two distinct phases, namely a tabu search phase and a probability learning based perturbation phase. To assess the effectiveness of the proposed algorithm, we show computational results on a set of 30 benchmark instances introduced in this paper and present comparative studies with respect to the approximation algorithm, the genetic algorithm and the CPLEX solver. [ABSTRACT FROM AUTHOR]

A unified distributed reinforcement learning (RL) solution is offered for both static and dynamic economic dispatch problems (EDPs). Each agent is assigned with a fixed, discrete, virtual action set, and a projection method generates the feasible, actual actions to satisfy the constraints. A distributed algorithm, based on singularly perturbed system, solves the projection problem. A distributed form of Hysteretic Q-learning achieves coordination among agents. Therein, the Q-values are developed based on the virtual actions, while the rewards are produced by the projected actual actions. The proposed algorithm deals with continuous action space and power loads without using function approximations. Theoretical analysis and comparative simulation studies verify algorithm's convergence and optimality. [ABSTRACT FROM AUTHOR]

We consider the problem of energy-efficient point-to-point transmission of delay-sensitive data (e.g., multimedia data) over a fading channel. We propose a rigorous and unified framework for simultaneously utilizing both physical-layer and system-level techniques to minimize energy consumption, under delay constraints, in the presence of stochastic and unknown traffic and channel conditions. We formulate the problem as a Markov decision process and solve it online using reinforcement learning. The advantages of the proposed online method are that i) it does not require a priori knowledge of the traffic arrival and channel statistics to determine the jointly optimal physical-layer and system-level power management strategies; ii) it exploits partial information about the system so that less information needs to be learned than when using conventional reinforcement learning algorithms; and iii) it obviates the need for action exploration, which severely limits the adaptation speed and run-time performance of conventional reinforcement learning algorithms. [ABSTRACT FROM AUTHOR]

Currently, industry is undergoing an exponential increase in binary-based combinatorial problems. In this regard, metaheuristics have been a common trend in the field in order to design approaches to successfully solve them. Thus, a well-known strategy includes the employment of continuous swarm-based algorithms transformed to perform in binary environments. In this work, we propose a hybrid approach that contains discrete smartly adapted population-based strategies to efficiently tackle binary-based problems. The proposed approach employs a reinforcement learning technique, known as SARSA (State–Action–Reward–State–Action), in order to utilize knowledge based on the run time. In order to test the viability and competitiveness of our proposal, we compare discrete state-of-the-art algorithms smartly assisted by SARSA. Finally, we illustrate interesting results where the proposed hybrid outperforms other approaches, thus, providing a novel option to tackle these types of problems in industry. [ABSTRACT FROM AUTHOR]

This study aims to solve the problems of poor exploration ability, single strategy, and high training cost in autonomous underwater vehicle (AUV) motion planning tasks and to overcome certain difficulties, such as multiple constraints and a sparse reward environment. In this research, an end-to-end motion planning system based on deep reinforcement learning is proposed to solve the motion planning problem of an underactuated AUV. The system directly maps the state information of the AUV and the environment into the control instructions of the AUV. The system is based on the soft actor–critic (SAC) algorithm, which enhances the exploration ability and robustness to the AUV environment. We also use the method of generative adversarial imitation learning (GAIL) to assist its training to overcome the problem that learning a policy for the first time is difficult and time-consuming in reinforcement learning. A comprehensive external reward function is then designed to help the AUV smoothly reach the target point, and the distance and time are optimized as much as possible. Finally, the end-to-end motion planning algorithm proposed in this research is tested and compared on the basis of the Unity simulation platform. Results show that the algorithm has an optimal decision-making ability during navigation, a shorter route, less time consumption, and a smoother trajectory. Moreover, GAIL can speed up the AUV training speed and minimize the training time without affecting the planning effect of the SAC algorithm. [ABSTRACT FROM AUTHOR]