Search

Reinforcement learning is one of the promising approaches for algorithmic trading in financial markets. However, in certain situations, buy or sell orders issued by an algorithmic trading program may not be fulfilled entirely. By considering the actual scenarios from the financial markets, in this paper, we propose a novel framework named Supervised Actor-Critic Reinforcement Learning with Action Feedback (SACRL-AF) for solving this problem. The action feedback mechanism of SACRL-AF notifies the actor about the dealt positions and corrects the transitions of the replay buffer. Meanwhile, the dealt positions are used as the labels for the supervised learning. Recent studies have shown that Deep Deterministic Policy Gradient (DDPG) and Twin Delayed Deep Deterministic Policy Gradient (TD3) are more stable and superior to other actor-critic algorithms. Against this background, based on the proposed SACRL-AF framework, two reinforcement learning algorithms henceforth referred to as Supervised Deep Deterministic Policy Gradient with Action Feedback (SDDPG-AF) and Supervised Twin Delayed Deep Deterministic Policy Gradient with Action Feedback (STD3-AF) are proposed in this paper. Experimental results show that SDDPG-AF and STD3-AF achieve the state-of-art performance in profitability. [ABSTRACT FROM AUTHOR]

In the order picking process of the warehouse center, considering the rapid increase in the volume of orders arriving at the picking center at the time of the promotional festival, a hybrid operation mode with multiple picking tables is used to meet the picking requirements of the huge number of real-time orders. Therefore, in this paper, a hybrid picking mode is proposed, taking into account both the idle degree of picking stations and their order item centers of gravity, and a new reinforcement learning algorithm embedding mechanism (PRL) with placeholder control is designed to solve the problem of a huge number of real-time item orders arriving at the picking center system on promotional holidays and in inconsistent quantities, and numerical simulations are performed for this algorithm. The experimental results show that the PRL algorithm in hybrid picking mode can handle a huge number of orders simultaneously and improve picking efficiency effectively. [ABSTRACT FROM AUTHOR]

Path planning of intelligent driving vehicles in emergencies is a hot research issue, this paper proposes a new method of the best path selection for the intelligent driving vehicles to solve this problem. Based on the prior knowledge applied reinforcement learning strategy and the searching- optimized A* algorithm, we designed a hybrid algorithm to help intelligent driving vehicles selecting the best path in the traffic network in emergencies including limited height, width, weight, accident, and traffic jam. Through simulation experiments and scene experiments, it is proved that the proposed algorithm has good stability, high efficiency, and practicability. [ABSTRACT FROM AUTHOR]

Oversampling is an effective method to fulfill imbalanced learning, owing to its easy-to-go capability of achieving the balance by synthesizing new samples. However, precise synthesizing in oversampling is always a significant yet challenging task due primarily to various problems such as noise samples, within-class imbalance, and selection of boundary samples. In order to solve these problems, this paper proposes a new improved oversampling method, called adaptively weighted three-way decision oversampling (AWTDO) for imbalanced learning. The working principle of the proposed AWTDO method includes three main steps. Firstly, remove the noise sample roughly, implement K-means clustering algorithm on raw data to establish multi-clusters, and calculate imbalanced ratio of each cluster. Secondly, classify all clusters into three categories according to their imbalanced ratios and three-way decision, such as positive domain, boundary domain, and negative domain. Accordingly, assign the number of synthetic samples distinguishably to each cluster regarding its category. Thirdly, determinatively select the target minority sample in each cluster and generate the new synthetic samples by using the stochastic linear interpolation technique according to different sampling weight. Finally, some comparative experiments on public datasets have shown that the proposed AWTDO method outperforms nine state-of-the-art oversampling methods. [ABSTRACT FROM AUTHOR]

As a complex nonlinear system, the first-order incremental relationship between the state variables of the beam and ball system (BABS) is asymmetric in the definition domain of the variables, and the characteristics of the system do not satisfy the superposition theorem. Studying the balance control of the BABS can help to better grasp the relevant characteristics of the nonlinear system. In this paper, the deep reinforcement learning method is used to study the BABS based on a visual sensor. First, the detail-reward function is designed by observing the control details of the system, and the rationality of the function is proved based on Q-function; secondly, considering and comparing the applicability of image processing methods in ball coordinate location, an intelligent location algorithm is proposed, and the location effects between the algorithms are compared and analyzed; then, combining the nonlinear theory and LQR theory, a reinforcement learning policy model is proposed to linearize near the equilibrium point, which significantly improves the control effect. Finally, experiments are designed to verify the effectiveness of the above methods in the control system. The experimental results show that the design scheme can be effectively applied to the control system of the BABS. It is verified that the introduction of detail-reward mechanism into a deep reinforcement learning algorithm can significantly reduce the complexity of the nonlinear control system and iterative algorithm, and effectively solve nonlinear control problems. [ABSTRACT FROM AUTHOR]

This paper presents an improved teaching-learning-based optimization (TLBO) algorithm for solving optimization problems, called RLTLBO. First, a new learning mode considering the effect of the teacher is presented. Second, the Q-Learning method in reinforcement learning (RL) is introduced to build a switching mechanism between two different learning modes in the learner phase. Finally, ROBL is adopted after both the teacher and learner phases to improve the local optima avoidance ability of RLTLBO. These two strategies effectively enhance the convergence speed and accuracy of the proposed algorithm. RLTLBO is analyzed on 23 standard benchmark functions and eight CEC2017 test functions to verify the optimization performance. The results reveal that proposed algorithm provides effective and efficient performance in solving benchmark test functions. Moreover, RLTLBO is also applied to solve eight industrial engineering design problems. Compared with the basic TLBO and seven state-of-the-art algorithms, the results illustrate that RLTLBO has superior performance and promising prospects for dealing with real-world optimization problems. The source codes of the RLTLBO are publicly available at https://github.com/WangShuang92/RLTLBO. [ABSTRACT FROM AUTHOR]

Based on the clinical states of the patient, dynamic treatment regime technology can provide various therapeutic methods, which is helpful for medical treatment policymaking. Reinforcement learning is an important approach for developing this technology. In order to implement the reinforcement learning algorithm efficiently, the computation of health data is usually outsourced to the untrustworthy cloud server. However, it may leak, falsify, or delete private health data. Encryption is a common method for solving this problem. But the cloud server is difficult to calculate encrypted health data. In this paper, based on Cheon et al.'s approximate homomorphic encryption scheme, we first propose secure computation protocols for implementing comparison, maximum, exponentiation, and division. Next, we design a homomorphic reciprocal of square root protocol firstly, which only needs one approximate computation. Based on the proposed secure computation protocols, we design a secure asynchronous advantage actor-critic reinforcement learning algorithm for the first time. Then, it is used to implement a secure treatment decision-making algorithm. Simulation results show that our secure computation protocols and algorithms are feasible. [ABSTRACT FROM AUTHOR]

Recent advancements in deep reinforcement learning (DRL) have led to its application in multi-agent scenarios to solve complex real-world problems, such as network resource allocation and sharing, network routing, and traffic signal controls. Multi-agent DRL (MADRL) enables multiple agents to interact with each other and with their operating environment, and learn without the need for external critics (or teachers), thereby solving complex problems. Significant performance enhancements brought about by the use of MADRL have been reported in multi-agent domains; for instance, it has been shown to provide higher quality of service (QoS) in network resource allocation and sharing. This paper presents a survey of MADRL models that have been proposed for various kinds of multi-agent domains, in a taxonomic approach that highlights various aspects of MADRL models and applications, including objectives, characteristics, challenges, applications, and performance measures. Furthermore, we present open issues and future directions of MADRL. [ABSTRACT FROM AUTHOR]

The reinforcement learning algorithms based on policy gradient may fall into local optimal due to gradient disappearance during the update process, which in turn affects the exploration ability of the reinforcement learning agent. In order to solve the above problem, in this paper, the cross-entropy method (CEM) in evolution policy, maximum mean difference (MMD), and twin delayed deep deterministic policy gradient algorithm (TD3) are combined to propose a diversity evolutionary policy deep reinforcement learning (DEPRL) algorithm. By using the maximum mean discrepancy as a measure of the distance between different policies, some of the policies in the population maximize the distance between them and the previous generation of policies while maximizing the cumulative return during the gradient update. Furthermore, combining the cumulative returns and the distance between policies as the fitness of the population encourages more diversity in the offspring policies, which in turn can reduce the risk of falling into local optimal due to the disappearance of the gradient. The results in the MuJoCo test environment show that DEPRL has achieved excellent performance on continuous control tasks; especially in the Ant-v2 environment, the return of DEPRL ultimately achieved a nearly 20% improvement compared to TD3. [ABSTRACT FROM AUTHOR]

Controlling a fleet of autonomous mobile robots (AMR) is a complex problem of optimization. Many approached have been conducted for solving this problem. They range from heuristics, which usually do not find an optimum, to mathematical models, which are limited due to their high computational effort. Machine Learning (ML) methods offer another potential trajectory for solving such complex problems. The focus of this brief survey is on Reinforcement Learning (RL) as a particular type of ML. Due to the reward-based optimization, RL offers a good basis for the control of fleets of AMR. In the context of this survey, different control approaches are investigated and the aspects of fleet control of AMR with respect to RL are evaluated. As a result, six fundamental key problems should be put on the current research agenda to enable a broader application in industry: (1) overcoming the "sim-to-real gap", (2) increasing the robustness of algorithms, (3) improving data efficiency, (4) integrating different fields of application, (5) enabling heterogeneous fleets with different types of AMR and (6) handling of deadlocks. [ABSTRACT FROM AUTHOR]

"Open world" environments are those in which novel objects, agents, events, and more can appear and contradict previous understandings of the environment. This runs counter to the "closed world" assumption used in most AI research, where the environment is assumed to be fully understood and unchanging. The types of environments AI agents can be deployed in are limited by the inability to handle the novelties that occur in open world environments. This paper presents a novel cognitive architecture framework to handle open-world novelties. This framework combines symbolic planning, counterfactual reasoning, reinforcement learning, and deep computer vision to detect and accommodate novelties. We introduce general algorithms for exploring open worlds using inference and machine learning methodologies to facilitate novelty accommodation. The ability to detect and accommodate novelties allows agents built on this framework to successfully complete tasks despite a variety of novel changes to the world. Both the framework components and the entire system are evaluated in Minecraft-like simulated environments. Our results indicate that agents are able to efficiently complete tasks while accommodating "concealed novelties" not shared with the architecture development team. [ABSTRACT FROM AUTHOR]

To solve the problems of poor exploration ability and convergence speed of traditional deep reinforcement learning in the navigation task of the patrol robot under indoor specified routes, an improved deep reinforcement learning algorithm based on Pan/Tilt/Zoom(PTZ) image information was proposed in this paper. The obtained symmetric image information and target position information are taken as the input of the network, the speed of the robot is taken as the output of the next action, and the circular route with boundary is taken as the test. The improved reward and punishment function is designed to improve the convergence speed of the algorithm and optimize the path so that the robot can plan a safer path while avoiding obstacles first. Compared with Deep Q Network(DQN) algorithm, the convergence speed after improvement is shortened by about 40%, and the loss function is more stable. [ABSTRACT FROM AUTHOR]

Multi-agent reinforcement learning (MARL) has been successfully applied in many fields. In MARL, the policy evaluation problem is one of crucial problems. In order to solve this problem, distributed Temporal-Difference (TD) learning algorithm is one of the most popular methods in a cooperative manner. Despite its empirical success, however, the theory of the adaptive variant of distributed TD learning still remain limited. To fill this gap, we propose an adaptive distributed temporal-difference algorithm (referred to as MS - ADTD) under Markovian sampling over time-varying networks. Furthermore, we rigorously analyze the convergence of MS - ADTD , the theoretical results show that the local estimation can converge linearly to the optimal neighborhood. Meanwhile, the theoretical results are verified by simulation experiments. • This paper proposes a distributed adaptive TD learning algorithm under Markovian sampling. • It theoretically analyzes the non-asymptotic convergence performance of the proposed algorithm. • It verifies the performance of the proposed algorithm by experiments. [ABSTRACT FROM AUTHOR]

Many real-world applications can be described as large-scale games of imperfect information, which require extensive prior domain knowledge, especially in competitive or human–AI cooperation settings. Population-based training methods have become a popular solution to learn robust policies without any prior knowledge, which can generalize to policies of other players or humans. In this survey, we shed light on population-based deep reinforcement learning (PB-DRL) algorithms, their applications, and general frameworks. We introduce several independent subject areas, including naive self-play, fictitious self-play, population-play, evolution-based training methods, and the policy-space response oracle family. These methods provide a variety of approaches to solving multi-agent problems and are useful in designing robust multi-agent reinforcement learning algorithms that can handle complex real-life situations. Finally, we discuss challenges and hot topics in PB-DRL algorithms. We hope that this brief survey can provide guidance and insights for researchers interested in PB-DRL algorithms. [ABSTRACT FROM AUTHOR]

The use of reinforcement learning technology for the optimal control problem solution is considered. To solve the optimal control problem an evolutionary algorithm is used that finds control to ensure the movements of a control object along different trajectories with approximately the same values of the quality criterion. Additional conditions for passing the trajectory in the neighbourhood of given areas of the state space are included in the quality criterion. To build a stabilization system for the movement of an object along a given trajectory, machine learning control by symbolic regression is used. An example of solving the optimal control problem for a quadcopter is given. [ABSTRACT FROM AUTHOR]

Video monitoring has a wide range of applications in a variety of scenarios, especially in smart parks. How to improve the efficiency of video data processing and reduce resource consumption have become of increasing concern. The high complexity of traditional computation offloading algorithms makes it difficult to apply them to real-time decision-making scenarios. Thus, we propose a multi-agent deep reinforcement learning algorithm with credit assignment (MACA) for computation offloading in smart park monitoring. By making online decisions after offline training, the agent can give consideration to both decision time and accuracy in effectively solving the problem of the curse of dimensionality. Via simulation, we compare the performance of MACA with traditional deep Q-network reinforcement learning algorithm and other methods. Our results show that MACA performs better in scenarios where there are a higher number of agents and can minimize request delay and reduce task energy consumption. In addition, we also provide results from a generalization capability verified experiment and ablation study, which demonstrate the contribution of MACA algorithm to each component. [ABSTRACT FROM AUTHOR]

Although many adaptive techniques for active vibration reduction have been designed to achieve optimal performance in practical applications, few are related to reinforcement learning (RL). To explore the best performance of the active vibration reduction system (AVRS) without prior knowledge, a self-adaptive parameter regulation method based on the DDPG algorithm was examined in this study. The DDPG algorithm is unsuitable for a random environment and prone to reward-hacking. To solve this problem, a reward function optimization method based on the integral area of the decibel (dB) value between transfer functions was investigated. Simulation and graphical experimental results show that the optimized DDPG algorithm can automatically track and maintain optimal control performance of the AVRS. [ABSTRACT FROM AUTHOR]

Connectivity is a critical prerequisite for the effective operation of data gathering and forwarding procedures in Wireless Sensor Networks (WSNs). Failure of several sensor nodes in the network makes the base station incapable of receiving data from all the portions of the target area. Multiple partitions that are unable to communicate with one another are formed. Such networks are repaired using additional mobile relays. These relays cooperatively work together to create links among the partitions. In order to complete their task, they require a quick, communication-efficient, and machine learning-based approach. Reinforcement learning has evolved as a very efficient approach with long-term solutions in solving such problems. In this work, we propose a Reinforcement Learning-based Connectivity Restoration (RLCR) method that applies an intelligent machine learning algorithm for collaborative movement and connection establishment among partitions using relay nodes. It takes into account the actions of other agents and is capable of learning complicated multi-agent coordinating strategies. In a subsequent step, relay selection and connectivity maintenance have also been included. In RLCR, relays search for isolated partitions while maintaining communication with one another. Besides, we use Python to simulate the procedure and compare the results to various current methodologies. The experimental analysis illustrates that the proposed RLCR method performs better than the standard algorithms. [ABSTRACT FROM AUTHOR]

Motivation Disease diagnosis-oriented dialog system models the interactive consultation procedure as the Markov decision process, and reinforcement learning algorithms are used to solve the problem. Existing approaches usually employ a flat policy structure that treat all symptoms and diseases equally for action making. This strategy works well in a simple scenario when the action space is small; however, its efficiency will be challenged in the real environment. Inspired by the offline consultation process, we propose to integrate a hierarchical policy structure of two levels into the dialog system for policy learning. The high-level policy consists of a master model that is responsible for triggering a low-level model, the low-level policy consists of several symptom checkers and a disease classifier. The proposed policy structure is capable to deal with diagnosis problem including large number of diseases and symptoms. Results Experimental results on three real-world datasets and a synthetic dataset demonstrate that our hierarchical framework achieves higher accuracy and symptom recall in disease diagnosis compared with existing systems. We construct a benchmark including datasets and implementation of existing algorithms to encourage follow-up researches. Availability and implementation The code and data are available from https://github.com/FudanDISC/DISCOpen-MedBox-DialoDiagnosis Supplementary information Supplementary data are available at Bioinformatics online. [ABSTRACT FROM AUTHOR]

Machine learning research has been able to solve problems in multiple domains. Machine learning represents an open area of research for solving optimisation problems. The optimisation problems can be solved using a metaheuristic algorithm, which can find a solution in a reasonable amount of time. However, the time required to find an appropriate metaheuristic algorithm, that would have the convenient configurations to solve a set of optimisation problems properly presents a problem. The proposal described in this article contemplates an approach that automatically creates metaheuristic algorithms given a set of optimisation problems. These metaheuristic algorithms are created by modifying their logical structure via the execution of an evolutionary process. This process employs an extension of the reinforcement learning approach that considers multi-agents in their environment, and a learning agent composed of an analysis process and a process of modification of the algorithms. The approach succeeded in creating a metaheuristic algorithm that managed to solve different continuous domain optimisation problems from the experiments performed. The implications of this work are immediate because they describe a basis for the generation of metaheuristic algorithms in an online-evolution. [ABSTRACT FROM AUTHOR]

The purpose of stock market investment is to obtain more profits. In recent years, an increasing number of researchers have tried to implement stock trading based on machine learning. Facing the complex stock market, how to obtain effective information from multisource data and implement dynamic trading strategies is difficult. To solve these problems, this study proposes a new deep reinforcement learning model to implement stock trading, analyzes the stock market through stock data, technical indicators and candlestick charts, and learns dynamic trading strategies. Fusing the features of different data sources extracted by the deep neural network as the state of the stock market, the agent in reinforcement learning makes trading decisions on this basis. Experiments on the Chinese stock market dataset and the S&P 500 stock market dataset show that our trading strategy can obtain higher profits compared with other trading strategies. [ABSTRACT FROM AUTHOR]

Traditional agent-based modelling is mostly rule-based. For many systems, this approach is extremely successful, since the rules are well understood. However, for a large class of systems it is difficult to find rules that adequately describe the behaviour of the agents. A simple example would be two agents playing chess: Here, it is impossible to find simple rules. To solve this problem, we introduce a framework for agent-based modelling that incorporates machine learning. In a process closely related to reinforcement learning, the agents learn rules. As a trade-off, a utility function needs to be defined, which is much simpler in most cases. We test this framework to replicate the results of the prominent Sugarscape model as a proof of principle. Furthermore, we investigate a more complicated version of the Sugarscape model, that exceeds the scope of the original framework. By expanding the framework we also find satisfying results there. [ABSTRACT FROM AUTHOR]

Routing is an essential part for network deployment to maintain and improve the network performance. With the rapid demands of various wireless applications, delay and energy efficiency are two fundamentally important aspects in the next-generation communication. A multi-objective Dyna-Q based routing (MODQR) approach to improve both delay and energy performances is proposed in this paper. For delay, the interference, asymmetrical link condition and probability of transmission failure (PTF) are considered. When deriving the PTF, the gray physical interference model is used. For energy, the ratio of consumed energy to energy capacity is used to monitor energy condition. Reinforcement learning is an effective way to solve the routing problem in a network with uncertain conditions. A path can be chosen by iterative exploration and exploitation. Dyna-Q is an effective reinforcement learning algorithm which can increase the convergence speed. To the best of our knowledge, Dyna-Q is used to solve the multi-objective routing problem for the first time. The path with least end-to-end delay and largest energy efficiency will be selected. Simulation results show that MODQR can obtain up to 80.97%, 83.48% and 86.15% better network performance than three other state-of-the-art routing methods. • A multi-objective routing approach considering delay and energy is proposed. • Delay is derived by considering multiple performance metrics on asymmetrical links. • Consumed energy and energy capacity are considered to evaluate energy status. • The improved Dyna-Q is used to solve the optimization model for the first time. • The performance is 80+% better than three existing state-of-the-art routing methods. [ABSTRACT FROM AUTHOR]

Currently, industry is undergoing an exponential increase in binary-based combinatorial problems. In this regard, metaheuristics have been a common trend in the field in order to design approaches to successfully solve them. Thus, a well-known strategy includes the employment of continuous swarm-based algorithms transformed to perform in binary environments. In this work, we propose a hybrid approach that contains discrete smartly adapted population-based strategies to efficiently tackle binary-based problems. The proposed approach employs a reinforcement learning technique, known as SARSA (State–Action–Reward–State–Action), in order to utilize knowledge based on the run time. In order to test the viability and competitiveness of our proposal, we compare discrete state-of-the-art algorithms smartly assisted by SARSA. Finally, we illustrate interesting results where the proposed hybrid outperforms other approaches, thus, providing a novel option to tackle these types of problems in industry. [ABSTRACT FROM AUTHOR]

The comprehensively completed BDS-3 short-message communication system, known as the short-message satellite communication system (SMSCS), will be widely used in traditional blind communication areas in the future. However, short-message processing resources for short-message satellites are relatively scarce. To improve the resource utilization of satellite systems and ensure the service quality of the short-message terminal is adequate, it is necessary to allocate and schedule short-message satellite processing resources in a multi-satellite coverage area. In order to solve the above problems, a short-message satellite resource allocation algorithm based on deep reinforcement learning (DRL-SRA) is proposed. First of all, using the characteristics of the SMSCS, a multi-objective joint optimization satellite resource allocation model is established to reduce short-message terminal path transmission loss, and achieve satellite load balancing and an adequate quality of service. Then, the number of input data dimensions is reduced using the region division strategy and a feature extraction network. The continuous spatial state is parameterized with a deep reinforcement learning algorithm based on the deep deterministic policy gradient (DDPG) framework. The simulation results show that the proposed algorithm can reduce the transmission loss of the short-message terminal path, improve the quality of service, and increase the resource utilization efficiency of the short-message satellite system while ensuring an appropriate satellite load balance. [ABSTRACT FROM AUTHOR]