Search

We study the quadratic cycle cover problem (QCCP), which aims to find a node-disjoint cycle cover in a directed graph with minimum interaction cost between successive arcs. We derive several semidefinite programming (SDP) relaxations and use facial reduction to make these strictly feasible. We investigate a nontrivial relationship between the transformation matrix used in the reduction and the structure of the graph, which is exploited in an efficient algorithm that constructs this matrix for any instance of the problem. To solve our relaxations, we propose an algorithm that incorporates an augmented Lagrangian method into a cutting-plane framework by utilizing Dykstra's projection algorithm. Our algorithm is suitable for solving SDP relaxations with a large number of cutting-planes. Computational results show that our SDP bounds and efficient cutting-plane algorithm outperform other QCCP bounding approaches from the literature. Finally, we provide several SDP-based upper bounding techniques, among which is a sequential Q-learning method that exploits a solution of our SDP relaxation within a reinforcement learning environment. Summary of Contribution: The quadratic cycle cover problem (QCCP) is the problem of finding a set of node-disjoint cycles covering all the nodes in a graph such that the total interaction cost between successive arcs is minimized. The QCCP has applications in many fields, among which are robotics, transportation, energy distribution networks, and automatic inspection. Besides this, the problem has a high theoretical relevance because of its close connection to the quadratic traveling salesman problem (QTSP). The QTSP has several applications, for example, in bioinformatics, and is considered to be among the most difficult combinatorial optimization problems nowadays. After removing the subtour elimination constraints, the QTSP boils down to the QCCP. Hence, an in-depth study of the QCCP also contributes to the construction of strong bounds for the QTSP. In this paper, we study the application of semidefinite programming (SDP) to obtain strong bounds for the QCCP. Our strongest SDP relaxation is very hard to solve by any SDP solver because of the large number of involved cutting-planes. Because of that, we propose a new approach in which an augmented Lagrangian method is incorporated into a cutting-plane framework by utilizing Dykstra's projection algorithm. We emphasize an efficient implementation of the method and perform an extensive computational study. This study shows that our method is able to handle a large number of cuts and that the resulting bounds are currently the best QCCP bounds in the literature. We also introduce several upper bounding techniques, among which is a distributed reinforcement learning algorithm that exploits our SDP relaxations. [ABSTRACT FROM AUTHOR]

This research paper presents the Buckley-James Q-learning (BJ-Q) algorithm, a cutting-edge method designed to optimize personalized treatment strategies, especially in the presence of right censoring. We critically assess the algorithm's effectiveness in improving patient outcomes and its resilience across various scenarios. Central to our approach is the innovative use of the survival time to impute the reward in Q-learning, employing the Buckley-James method for enhanced accuracy and reliability. Our findings highlight the significant potential of personalized treatment regimens and introduce the BJ-Q learning algorithm as a viable and promising approach. This work marks a substantial advancement in our comprehension of treatment dynamics and offers valuable insights for augmenting patient care in the ever-evolving clinical landscape. [ABSTRACT FROM AUTHOR]

In this paper, a fractional-order control method based on the twin-delayed deep deterministic policy gradient (TD3) algorithm in reinforcement learning is proposed. A fractional-order disturbance observer is designed to estimate the disturbances, and the radial basis function network is selected to approximate system uncertainties in the system. Then, a fractional-order sliding-mode controller is constructed to control the system, and the parameters of the controller are tuned using the TD3 algorithm, which can optimize the control effect. The results show that the fractional-order control method based on the TD3 algorithm can not only improve the closed-loop system performance under different operating conditions but also enhance the signal tracking capability. [ABSTRACT FROM AUTHOR]

The capacitated dispersion problem, which is a variant of the maximum diversity problem, aims to determine a set of elements within a network. These elements could symbolize, for instance, facilities in a supply chain or transmission nodes in a telecommunication network. While each element typically has a bounded service capacity, in this research, we introduce a twist. The capacity of each node might be influenced by a random Bernoulli component, thereby rendering the possibility of a node having zero capacity, which is contingent upon a black box mechanism that accounts for environmental variables. Recognizing the inherent complexity and the NP-hard nature of the capacitated dispersion problem, heuristic algorithms have become indispensable for handling larger instances. In this paper, we introduce a novel approach by hybridizing a heuristic algorithm with reinforcement learning to address this intricate problem variant. [ABSTRACT FROM AUTHOR]

In recent years, the utilization of dual-arm robots has gained substantial prominence across various industries owing to their collaborative operational capabilities. In order to achieve collision avoidance and facilitate cooperative task completion, efficient path planning plays a pivotal role. The high dimensionality associated with collaborative task execution in dual-arm robots renders existing path planning methods ineffective for conducting efficient exploration. This paper introduces a multi-agent path planning reinforcement learning algorithm that integrates an experience replay strategy, a shortest-path constraint, and the policy gradient method. To foster collaboration and avoid competition between the robot arms, the proposed approach incorporates a mechanism known as "reward cooperation, punishment competition" during the training process. Our algorithm demonstrates strong performance in the control of dual-arm robots and exhibits the potential to mitigate the challenge of reward sparsity encountered during the training process. The effectiveness of the proposed algorithm is validated through simulations and experiments, comparing the results with existing methods and showcasing its superiority in dual-arm robot path planning. [ABSTRACT FROM AUTHOR]

Reinforcement learning algorithms often require finiteness of state and action spaces in Markov decision processes (MDPs) (also called controlled Markov chains) and various efforts have been made in the literature towards the applicability of such algorithms for continuous state and action spaces. In this paper, we show that under very mild regularity conditions (in particular, involving only weak continuity of the transition kernel of an MDP), Q-learning for standard Borel MDPs via quantization of states and actions (called Quantized Q-Learning) converges to a limit, and furthermore this limit satisfies an optimality equation which leads to near optimality with either explicit performance bounds or which are guaranteed to be asymptotically optimal. Our approach builds on (i) viewing quantization as a measurement kernel and thus a quantized MDP as a partially observed Markov decision process (POMDP), (ii) utilizing near optimality and convergence results of Q-learning for POMDPs, and (iii) finally, near-optimality of finite state model approximations for MDPs with weakly continuous kernels which we show to correspond to the fixed point of the constructed POMDP. Thus, our paper presents a very general convergence and approximation result for the applicability of Q-learning for continuous MDPs. [ABSTRACT FROM AUTHOR]

With the accelerated development of smart cities, the concept of a "smart industrial park" in which unmanned ground vehicles (UGVs) have wide application has entered the industrial field of vision. When faced with multiple tasks and heterogeneous tasks, the task execution efficiency of a single UGV is inefficient, thus the task planning research under multi-UGV cooperation has become more urgent. In this paper, under the anti-collision cooperation mechanism for multi-UGV path planning, an improved algorithm with optimized-weighted-speedy Q-learning (OWS Q-learning) is proposed. The slow convergence speed of the Q-learning algorithm is overcome to a certain extent by changing the update mode of the Q function. By improving the selection mode of learning rate and the selection strategy of action, the relationship between exploration and utilization is balanced, and the learning efficiency of multi-agent in complex environments is improved. The simulation experiments in static environment show that the designed anti-collision coordination mechanism effectively solves the coordination problem of multiple UGVs in the same scenario. In the same experimental scenario, compared with the Q-learning algorithm and other reinforcement learning algorithms, only the OWS Q-learning algorithm achieves the convergence effect, and the OWS Q-learning algorithm has the shortest collision-free path for UGVS and the least time to complete the planning. Compared with the Q-learning algorithm, the calculation time of the OWS Q-learning algorithm in the three experimental scenarios is improved by 53.93%, 67.21%, and 53.53%, respectively. This effectively improves the intelligent development of UGV in smart parks. [ABSTRACT FROM AUTHOR]

Compressive data gathering (CDG) is an adequate method to reduce the amount of data transmission, thereby decreasing energy expenditure for wireless sensor networks (WSNs). Sleep scheduling integrated with CDG can further promote energy efficiency. Most of existing sleep scheduling methods for CDG were formulated as centralized optimization problems which introduced many extra control message exchanges. Meanwhile, a few distributed methods usually adopted stochastic decision which could not adapt to variance in residual energy of nodes. A part of nodes were prone to prematurely run out of energy. In this paper, a reinforcement learning-based sleep scheduling algorithm for CDG (RLSSA-CDG) is proposed. Active nodes selection is modeled as a finite Markov decision process. The mode-free Q learning algorithm is used to search optimal decision strategies. Residual energy of nodes and sampling uniformity are considered into the reward function of the Q learning algorithm for load balance of energy consumption and accurate data reconstruction. It is a distributed algorithm that avoids large amounts of control message exchanges. Each node takes part in one step of the decision process. Thus, computation overhead for sensor nodes is affordable. Simulation experiments are carried out on the MATLAB platform to validate the effectiveness of the proposed RLSSA-CDG against the distributed random sleep scheduling algorithm for CDG (DSSA-CDG) and the original sparse-CDG algorithm without sleep scheduling. The simulation results indicate that the proposed RLSSA-CDG outperforms the two contrast algorithms in terms of energy consumption, network lifetime, and data recovery accuracy. The proposed RLSSA-CDG reduces energy consumption by 4.64% and 42.42%, respectively, compared to the DSSA-CDG and the original sparse-CDG, prolongs life span by 57.3%, and promotes data recovery accuracy by 84.7% compared to the DSSA-CDG. [ABSTRACT FROM AUTHOR]

This paper presents a new approach to finding optimal patterns for the placement of fields of sonobuoys in a complex undersea environment. We model the problem as a biobjective one, where the aim is to minimise both sensor placement time and uncertainty over target localisation. Both objectives may be important in time‐critical localisation scenarios and our approach allows an operator to choose between different optimal solutions, favouring lower placement time or lower localisation uncertainty as operational circumstances require. We develop a two‐phase algorithm, where an offline multiobjective evolutionary phase finds initial Pareto‐non‐dominated solutions to a static problem and then an online multiobjective reinforcement learning phase finds improved solutions using updated information. We find that the evolutionary algorithm improves significantly on standard grid patterns and that the reinforcement learning algorithm improves further on the evolutionary phase. The number of sonobuoys required may also be reduced. [ABSTRACT FROM AUTHOR]

Aiming at the problem that unmanned combat aerial vehicle (UCAV) is difficult to quickly and accurately perceive situation information and make maneuvering decision autonomously in modern air combat, which is easily affected by complex factors, a maneuvering decision algorithm of UCAV combined with deep reinforcement learning and game theory is proposed in this paper. Firstly, through the UCAV dynamics model and maneuver library, a reasonable air combat situation assessment model and advantage reward function are established, and the sample data of situation assessment indicators are constructed using the structure entropy weight method. Secondly, the convolutional neural network (CNN) is used to process the high-dimensional continuous situation features of UCAV in air combat, eliminate the correlation and redundancy between situation features, and train the neural network to approximate the action-value function. Then, the double deep Q network (DDQN) algorithm in reinforcement learning (RL) is introduced to train the agent by the interaction with the environment and combined with Minimax algorithm in stochastic game theory to solve the optimal value function in each specific state, and the optimal maneuver decision of UCAV is obtained. Air combat simulation results show that UCAV can choose maneuvers autonomously under different situations and occupy a dominant position quickly by this method, which greatly improves the combat effectiveness of UCAV. [ABSTRACT FROM AUTHOR]

Q-learning is a classical reinforcement learning algorithm and one of the most important methods of mobile robot path planning without a prior environmental model. Nevertheless, Q-learning is too simple when initializing Q-table and wastes too much time in the exploration process, causing a slow convergence speed. This paper proposes a new Q-learning algorithm called the Paired Whale Optimization Q-learning Algorithm (PWOQLA) which includes four improvements. Firstly, to accelerate the convergence speed of Q-learning, a whale optimization algorithm is used to initialize the values of a Q-table. Before the exploration process, a Q-table which contains previous experience is learned to improve algorithm efficiency. Secondly, to improve the local exploitation capability of the whale optimization algorithm, a paired whale optimization algorithm is proposed in combination with a pairing strategy to speed up the search for prey. Thirdly, to improve the exploration efficiency of Q-learning and reduce the number of useless explorations, a new selective exploration strategy is introduced which considers the relationship between current position and target position. Fourthly, in order to balance the exploration and exploitation capabilities of Q-learning so that it focuses on exploration in the early stage and on exploitation in the later stage, a nonlinear function is designed which changes the value of ε in ε-greedy Q-learning dynamically based on the number of iterations. Comparing the performance of PWOQLA with other path planning algorithms, experimental results demonstrate that PWOQLA achieves a higher level of accuracy and a faster convergence speed than existing counterparts in mobile robot path planning. The code will be released at https://github.com/wanghanyu0526/improveQL.git. [ABSTRACT FROM AUTHOR]

The deep Q-network (DQN) is one of the most successful reinforcement learning algorithms, but it has some drawbacks such as slow convergence and instability. In contrast, the traditional reinforcement learning algorithms with linear function approximation usually have faster convergence and better stability, although they easily suffer from the curse of dimensionality. In recent years, many improvements to DQN have been made, but they seldom make use of the advantage of traditional algorithms to improve DQN. In this paper, we propose a novel Q-learning algorithm with linear function approximation, called the minibatch recursive least squares Q-learning (MRLS-Q). Different from the traditional Q-learning algorithm with linear function approximation, the learning mechanism and model structure of MRLS-Q are more similar to those of DQNs with only one input layer and one linear output layer. It uses the experience replay and the minibatch training mode and uses the agent's states rather than the agent's state-action pairs as the inputs. As a result, it can be used alone for low-dimensional problems and can be seamlessly integrated into DQN as the last layer for high-dimensional problems as well. In addition, MRLS-Q uses our proposed average RLS optimization technique, so that it can achieve better convergence performance whether it is used alone or integrated with DQN. At the end of this paper, we demonstrate the effectiveness of MRLS-Q on the CartPole problem and four Atari games and investigate the influences of its hyperparameters experimentally. [ABSTRACT FROM AUTHOR]

This paper studies an unmanned aerial vehicle (UAV)-assisted wireless powered IoT network, where a rotary-wing UAV adopts fly-hover-communicate protocol to successively visit IoT devices in demand. During the hovering periods, the UAV works on full-duplex mode to simultaneously collect data from the target device and charge other devices within its coverage. Practical propulsion power consumption model and non-linear energy harvesting model are taken into account. We formulate a multi-objective optimization problem to jointly optimize three objectives: maximization of sum data rate, maximization of total harvested energy and minimization of UAV’s energy consumption over a particular mission period. These three objectives are in conflict with each other partly and weight parameters are given to describe associated importance. Since IoT devices keep gathering information from the physical surrounding environment and their requirements to upload data change dynamically, online path planning of the UAV is required. In this paper, we apply deep reinforcement learning algorithm to achieve online decision. An extended deep deterministic policy gradient (DDPG) algorithm is proposed to learn control policies of UAV over multiple objectives. While training, the agent learns to produce optimal policies under given weights conditions on the basis of achieving timely data collection according to the requirement priority and avoiding devices’ data overflow. The verification results show that the proposed MODDPG (multi-objective DDPG) algorithm achieves joint optimization of three objectives and optimal policies can be adjusted according to weight parameters among optimization objectives. [ABSTRACT FROM AUTHOR]

The doubly fed induction generator (DFIG) usually experiences high rotor current and DC capacitor link voltage spikes during system fault events. In this paper, a novel data‐driven approach is proposed to enhance DFIG performance under fault scenarios. An advanced reinforcement learning algorithm called guided surrogate‐gradient‐based evolution strategy (GSES) is used to control the DFIG power and capacitor DC‐link voltage by adjusting the optimal reference signals. This controller is able to prevent the DFIG rotor from over‐current risk and maintain grid‐connected operation. The proposed GSES‐based control algorithm was evaluated through simulations on a 3.6‐MW DFIG in the PSCAD/EMTDC software. Results have validated the effectiveness of the proposed GSES‐based control algorithm in improving DFIG performance under various fault scenarios. [ABSTRACT FROM AUTHOR]

The lighting requirements are subjective and one light setting cannot work for all. However, there is little work on developing smart lighting algorithms that can adapt to user preferences. To address this gap, this paper uses fuzzy logic and reinforcement learning to develop an adaptive lighting algorithm. In particular, we develop a baseline fuzzy inference system (FIS) using the domain knowledge, generating light recommendation based on a set of intuitive rules. These rules, derived from existing literature, are based on environmental conditions i.e. daily glare index, and user information including age, activity, and chronotype. Through a feedback mechanism, the user interacts with the algorithm, correcting the algorithm output to their preferences. We interpret these corrections as rewards to a Q-learning algorithm, which tunes the FIS parameters online to match the user preferences. The Q-learning is a model-free learning algorithm that learns to act optimally by interacting with the user and the rewards it receives. This allows the proposed algorithm to work in a model-free manner, effectively handling the uncertainties that might arise from the individualistic preferences of users. To the authors' best knowledge, this algorithm is pioneering work in designing intelligent algorithms for personalized lighting control, featuring several elements of novelty, including the number of environmental and user-related inputs, the continuous control of light intensity as opposed to common on/off control, and the ability to learn user preferences. The algorithm is implemented in a real aircraft cabin and is evaluated in an extensive user study. The implementation results demonstrate that the developed algorithm possesses the capability to learn user preferences while successfully adapting to a wide range of environmental conditions and user characteristics. This underscores its viability as a potent solution for intelligent light management, featuring advanced learning capabilities. • Intelligent lighting algorithm with the ability to learn from and adapt to user preferences. • A fuzzy inference system tunes lighting based on environment and user traits. • The developed algorithm is tested and verified via an in-depth user study. [ABSTRACT FROM AUTHOR]

Grid-interactive building control poses a critical challenge in the context of grid modernization and decarbonization. Recently, various artificial intelligence-based optimal control approaches have been proposed, providing innovative solutions to optimal building control problems. However, researchers and practitioners are now confronted with a dilemma when selecting appropriate control strategies, ranging from model-based (knowledge-incorporating) to learning-based (data-driven) approaches, and hybrid methods combining them. Although each algorithm in existing literature claims superiority over specific baselines, their performance has never been systematically compared and analyzed, owing to the absence of a unified platform for comprehensive evaluation. To fill this knowledge gap, we identify and implement all state-of-the-art approaches within a modular training and evaluation framework, assessing their efficacy in a grid-interactive building control problem. In this paper, we also introduce a streamlined hybrid method that complements existing hybrid approaches. Our comparative study reveals and quantifies the advantages of hybrid methods: on average, they achieve near-optimal control while requiring less than 14% of the online computation of traditional model-based methods. To achieve this performance, they need as few as 2% of the training samples of purely learning-based methods. Finally, we provide insights into the merits, limitations, and implementation of each method to help researchers better understand the state of the art and future directions. The software framework implemented in this study is open-sourced to facilitate future research. [ABSTRACT FROM AUTHOR]

In this paper, we present a design method for a wideband non-uniformly spaced linear array (NUSLA), with both symmetric and asymmetric geometries, using the modified reinforcement learning algorithm (MORELA). We designed a cost function that provided freedom to the beam pattern by setting limits only on the beam width (BW) and side-lobe level (SLL) in order to satisfy the desired BW and SLL in the wide band. We added the scan angle condition to the cost function to design the scanned beam pattern, as the ability to scan a beam in the desired direction is important in various applications. In order to prevent possible pointing angle errors for asymmetric NUSLA, we employed a penalty function to ensure the peak at the desired direction. Modified reinforcement learning algorithm (MORELA), which is a reinforcement learning-based algorithm used to determine a global optimum of the cost function, is applied to optimize the spacing and weights of the NUSLA by minimizing the proposed cost function. The performance of the proposed scheme was verified by comparing it with that of existing heuristic optimization algorithms via computer simulations. [ABSTRACT FROM AUTHOR]

For vehicle-to-network communications, handover (HO) management enables vehicles to maintain the connection with the network while transiting through coverage areas of different base stations (BSs). However, the high mobility of vehicles means shorter connection periods with each BS that leads to frequent HOs, hence raises the necessity for optimal HO decision making for high quality infotainment services. Machine learning is capable of capturing underlying pattern via data driven methods to find optimal solutions to complex problems, and much learning-based HO optimization research has been conducted focusing on specific network setups. However, attention still needs to be paid to the actual deployment aspect and standardized datasets or simulation environments for evaluation. This paper proposes a deep reinforcement learning-based HO algorithm using the input parameters that are configurable in the existing measurement report of cellular networks. The performance of the proposed algorithm is evaluated using the well-known network simulator ns-3 with its official LTE module. A realistic network setup in the city center of Glasgow (U.K.) is configured with vehicle trajectories generated by the routes mobility model using Google Maps Directions API. Evaluation results reveal that the proposed algorithm significantly outperforms the A3 RSRP baseline with an average of 25.72% packet loss reduction per HO, suggesting significant improvement in quality of service of phone call and video streaming, etc. The proposed algorithm also has a small implementation cost compared to some state-of-the-art and should be deployed by a software update to a local BS controller. [ABSTRACT FROM AUTHOR]

In the order picking process of the warehouse center, considering the rapid increase in the volume of orders arriving at the picking center at the time of the promotional festival, a hybrid operation mode with multiple picking tables is used to meet the picking requirements of the huge number of real-time orders. Therefore, in this paper, a hybrid picking mode is proposed, taking into account both the idle degree of picking stations and their order item centers of gravity, and a new reinforcement learning algorithm embedding mechanism (PRL) with placeholder control is designed to solve the problem of a huge number of real-time item orders arriving at the picking center system on promotional holidays and in inconsistent quantities, and numerical simulations are performed for this algorithm. The experimental results show that the PRL algorithm in hybrid picking mode can handle a huge number of orders simultaneously and improve picking efficiency effectively. [ABSTRACT FROM AUTHOR]

This review presents a high-level overview of the uses of machine learning (ML) to address several challenges in spatial auditory display research, primarily using head-related transfer functions. This survey also reviews and compares several categories of ML techniques and their application to virtual auditory reality research. This work addresses the use of ML techniques such as dimensionality reduction, unsupervised learning, supervised learning, reinforcement learning, and deep learning algorithms. The paper concludes with a discussion of the usage of ML algorithms to address specific spatial auditory display research challenges. [ABSTRACT FROM AUTHOR]

• The AES power system intelligent energy scheduling environment is built. • The BI-LSTM-Att action network is propose by combining BI-LSTM with attention mechanism. • The DQN-CE algorithm is proposed by improving the DQN algorithm. • The experiments on an all-electric ferry is conducted using a variety of methods. At present, Ship energy scheduling based on deep reinforcement learning (DRL) is an important research direction in which Deep Q learning algorithm (DQN) has been successfully applied in the field of energy scheduling. In terms of the shortcomings of DQN in all electric ships(AES) energy scheduling, such that, the insufficient performance of using multiplayer perceptron(MLP) as action network and the degradation of convergence performance due to over estimation of Q value, in this paper, we propose a DQN cross entropy(DQN-CE) energy scheduling algorithm combing bi-directional LSTM and attention mechanism(BI-LSTM-Att). Instead of using MLP, firstly, we use BI-LSTM-Att as action network of DQN, and then an improved DRL algorithm called DQN-CE is proposed, that is, we add cross-entropy loss of predicted action of the next time for the target network and the action network to DQN. Finally, we conduct simulation experiments on an all-electric ferry, and the experimental results show that the energy scheduling algorithm proposed in this paper reduces the economic consumption by 4.11%, increases the utilization rate of energy storage system by 24.4%, and reduces the exploration time of agent training by 31.3% compared with the original DQN energy scheduling algorithm. Finally, the effectiveness and superiority of the algorithm were further verified in the new case study. [ABSTRACT FROM AUTHOR]

The reinforcement learning algorithms based on policy gradient may fall into local optimal due to gradient disappearance during the update process, which in turn affects the exploration ability of the reinforcement learning agent. In order to solve the above problem, in this paper, the cross-entropy method (CEM) in evolution policy, maximum mean difference (MMD), and twin delayed deep deterministic policy gradient algorithm (TD3) are combined to propose a diversity evolutionary policy deep reinforcement learning (DEPRL) algorithm. By using the maximum mean discrepancy as a measure of the distance between different policies, some of the policies in the population maximize the distance between them and the previous generation of policies while maximizing the cumulative return during the gradient update. Furthermore, combining the cumulative returns and the distance between policies as the fitness of the population encourages more diversity in the offspring policies, which in turn can reduce the risk of falling into local optimal due to the disappearance of the gradient. The results in the MuJoCo test environment show that DEPRL has achieved excellent performance on continuous control tasks; especially in the Ant-v2 environment, the return of DEPRL ultimately achieved a nearly 20% improvement compared to TD3. [ABSTRACT FROM AUTHOR]

Autonomous vehicles or self-driving cars are prevalent nowadays, many vehicle manufacturers, and other tech companies are trying to develop autonomous vehicles. One major goal of the self-driving algorithms is to perform manoeuvres safely, even when some anomaly arises. To solve these kinds of complex issues, Artificial Intelligence and Machine Learning methods are used. One of these motion planning problems is when the tires lose their grip on the road, an autonomous vehicle should handle this situation. Thus the paper provides an Autonomous Drifting algorithm using Reinforcement Learning. The algorithm is based on a model-free learning algorithm, Twin Delayed Deep Deterministic Policy Gradients (TD3). The model is trained on six different tracks in a simulator, which is developed specifically for autonomous driving systems; namely CARLA. [ABSTRACT FROM AUTHOR]

• The paper proposes a new algorithm for reinforcement learning for solving optimal control problems. • An application of biodiesel reactor optimization problem is used to illustrate the application of RL to optimal control problems. • The advantages and disadvantages of RL for solving optimal control problems are illustrated with a real-world application. Optimal control problems are one of the most challenging problems in optimization. This paper presents a new and efficient Reinforcement Learning approach to optimal control problems based on the Batch Q-learning algorithm. To improve the convergence of the RL algorithm, we use k-dimensional uniformity of advanced sampling procedures, namely employing Hamersley sequences (HSS). HSS is used to randomly sample the state variables and discrete controls from the action space for the RL optimal control problem. The Neural-fitted Q-iterative algorithm is applied to solve an optimal control problem for a first-order state dynamical system. A real-world application of optimal temperature profile determination for biodiesel production in a batch reactor is presented. We present the comparison of our HSS-RL algorithm with that of the maximum principle. [ABSTRACT FROM AUTHOR]

Aeroengine fault detection is an important means to ensure flight safety. The application premise of data driven fault detection method is that all data come from the same distribution. However, this assumption is invalid in the actual engine fault detection, because the engine state will change with the increase of operating time, and the collected data will also have distribution differences. Due to the high cost of collecting engine data, it is difficult to collect enough data from the current state. Fortunately, transfer learning can transfer data information from other fields to the target field, thus alleviating the problem of data scarcity in the target domain. Therefore, from the idea of transfer learning, this paper proposes a cross domain aeroengine fault detection method, viz. transfer learning based on kernel perception algorithm with uneven margins (TL-KPAUM). The proposed method is divided into two stages. In the first stage, KPAUM is trained using the data in the source domain to extract the information in the source domain. In the second stage, the data in the target domain is used to realize the adaptation to the target domain. Compared with several baseline methods, TL-KPAUM has better performance when the amount of data in the target domain is small. Finally, using the simulation data of turbofan engine, the fault detection experiments of degraded state are designed, and the results show that this method is effective. [Display omitted] • A transfer learning algorithm is proposed for aeroengine fault detection. • The proposed method is applied to engine fault detection in different domains. • Simulation results show the effectiveness of this method compared with other methods. • The proposed method has good performance in the case of sparse data in the target domain. [ABSTRACT FROM AUTHOR]

In multi-user downlink small cell networks, cooperative resource allocation (RA) within a small cell cluster is a key technique to enhance network capacity. However, capacity-maximizing RA in frequency-selective fading channels requires global channel state information (CSI) of users within a small cell cluster, which makes it infeasible in practical networks with limited direct link capacity. To circumvent this global CSI assumption, most of the existing studies on RA have been based on several CSI assumptions such as local CSI and local CSI at the transmitters (CSIT). Nevertheless, cost functions with local CSI or local CSIT in the literature rely on heuristic formulations, because the sum-rate cannot be computed if without global CSI. In this paper, we propose a deep reinforcement learning-based RA algorithm to maximize the sum-rate for any given limited information on instantaneous CSI or sum-rate at the previous period. The proposed scheme is not restricted to certain CSI assumptions, but attempts to find the best RA for any given information such as quantized local CSI and quantized local CSIT; thus, it is applicable to any given direct link capacity. The proposed algorithm is self-adaptive in time-varying channels, since it is not divided into training and test phases. We modify the target neural network (TNN) scheme to enhance the sum-rate and the convergence speed. Numerical simulations confirm that: i) the proposed algorithm outperforms the conventional algorithms even under the same CSI assumption such as local CSI and local CSIT; ii) a flexible trade-off between the amount of CSI and the sum-rate is realizable in practical systems. [ABSTRACT FROM AUTHOR]

The free-flying space manipulators present challenges in controlling the motions of both the spacecraft bus and the manipulator, because of the highly-coupling system dynamics and the unknown space environment disturbances. Although the sliding mode controllers are robust to the unknown disturbances and system uncertainties, the chattering effect could affect the pointing accuracy and the lifetime of the actuators. This paper first introduces the dynamics of a CuBot, which is a 3-rigid-link manipulator based on the CubeSat platform. To maintain the robustness while decreasing the chattering effect, an innovative reinforcement learning based fuzzy adaptive sliding mode controller is proposed. To maintain the robustness while reducing the chattering effect, an innovative reinforcement learning based fuzzy adaptive sliding mode controller is proposed. The switching gain is updated to estimate the lumped upper bound of the system uncertainties and the unknown disturbances, and then a new fuzzy logic adaptive law is applied on the switching gain to decrease the chattering effects. On top of that, the fuzzy logic rules are tuned by an innovative modified reinforcement learning mechanism to achieve the better tracking performance. The uniformly ultimately bounded tracking errors are guaranteed by the proposed control scheme, and the effectiveness is validated by the simulation results. • A space manipulator is designed on the 8U CubeSat bus. • A modified reinforcement learning algorithm is designed to tune the fuzzy logic rules. • A fuzzy adaptive sliding mode controller is designed to reduce the chattering effect. • The new algorithm is compared to a conventional sliding mode controller. • The performance of the new algorithm is verified by simulation. [ABSTRACT FROM AUTHOR]

In this paper, a deep reinforcement learning-based robust control strategy for quadrotor helicopters is proposed. The quadrotor is controlled by a learned neural network which directly maps the system states to control commands in an end-to-end style. The learning algorithm is developed based on the deterministic policy gradient algorithm. By introducing an integral compensator to the actor-critic structure, the tracking accuracy and robustness have been greatly enhanced. Moreover, a two-phase learning protocol which includes both offline and online learning phase is proposed for practical implementation. An offline policy is first learned based on a simplified quadrotor model. Then, the policy is online optimized in actual flight. The proposed approach is evaluated in the flight simulator. The results demonstrate that the offline learned policy is highly robust to model errors and external disturbances. It also shows that the online learning could significantly improve the control performance. [ABSTRACT FROM AUTHOR]

In this paper, we study a real-time monitoring system in which multiple source nodes are responsible for sending update packets to a common destination node in order to maintain the freshness of information at the destination. Since it may not always be feasible to replace or recharge batteries in all source nodes, we consider that the nodes are powered through wireless energy transfer (WET) by the destination. For this system setup, we investigate the optimal online sampling policy (referred to as the age-optimal policy) that jointly optimizes WET and scheduling of update packet transmissions with the objective of minimizing the long-term average weighted sum of Age of Information (AoI) values for different physical processes (observed by the source nodes) at the destination node, referred to as the sum-AoI. To solve this optimization problem, we first model this setup as an average cost Markov decision process (MDP) with finite state and action spaces. Due to the extreme curse of dimensionality in the state space of the formulated MDP, classical reinforcement learning algorithms are no longer applicable to our problem even for reasonable-scale settings. Motivated by this, we propose a deep reinforcement learning (DRL) algorithm that can learn the age-optimal policy in a computationally-efficient manner. We further characterize the structural properties of the age-optimal policy analytically, and demonstrate that it has a threshold-based structure with respect to the AoI values for different processes. We extend our analysis to characterize the structural properties of the policy that maximizes average throughput for our system setup, referred to as the throughput-optimal policy. Afterwards, we analytically demonstrate that the structures of the age-optimal and throughput-optimal policies are different. We also numerically demonstrate these structures as well as the impact of system design parameters on the optimal achievable average weighted sum-AoI. [ABSTRACT FROM AUTHOR]

Long-Term Evolution (LTE) over unlicensed spectrum extends LTE technology to the spacious unlicensed spectrum with readily available bandwidth. The provided capacity surge makes it one of the most high-profile technologies to meet the explosive growth of mobile traffic demand. Among its different variants, Licensed Assisted Access (LAA) is considered as a promising global solution attributed to its mandatory listen-before-talk (LBT) procedure. Nevertheless, although LBT effectively maintains transmission fairness between LTE and other unlicensed systems (e.g., Wi-Fi), the current LAA protocol specified in 3GPP Release 13 is far from perfect to achieve harmony coexistence. To this end, in this paper, we first develop an analytical model to evaluate the throughput performance of Category 4 (Cat 4) algorithm agreed in 3GPP release 13. Subject to the system fairness constraint, the aggregate throughput of LTE-LAA and Wi-Fi networks is maximized based on a semi branch and bound algorithm. To make the complex optimization tractable, reinforcement learning techniques are introduced to intelligently tune the contention window size for both LTE-LAA and Wi-Fi nodes. Specifically, a cooperative learning algorithm is developed assuming that the information between different systems is exchangeable. A non-cooperative version is subsequently developed to remove the previous assumption for better practicability. Extensive simulations are conducted to demonstrate the performance of the proposed learning algorithms in contrast to the analytical upper bound under various conditions. It is shown that both proposed learning algorithms can significantly improve the total throughput performance while satisfying the fairness constraints. Particularly, the proposed cooperative learning algorithm can closely approach the analytical bound. [ABSTRACT FROM AUTHOR]

The paper proposes to apply an algorithm for predicting the minimum level of the state of charge (SoC) of stationary supercapacitor energy storage system operating in a DC traction substation, and for changing it over time. This is done to insure maximum energy recovery for trains while braking. The model of a supercapacitor energy storage system, its algorithms of operation and prediction of the minimum state of charge are described in detail; the main formulae, graphs and results of simulation are also provided. It is proposed to divide the SoC curve into equal periods of time during which the minimum states of charge remain constant. To predict the SoC level for the subsequent period, the learning algorithm based on the neural network could be used. Then, the minimum SoC could be based on two basic types of data: the first one is the time profile of the energy storage load during the previous period with the constant minimum SoC retained, while the second one relies on the trains' locations and speed values in the previous period. It is proved that the use of variable minimum SoC ensures an increase of the energy volume recovered by approximately 10%. Optimum architecture and activation function of the neural network are also found. [ABSTRACT FROM AUTHOR]

In the current unmanned aircraft systems (UASs) for sensing services, unmanned aerial vehicles (UAVs) transmit their sensory data to terrestrial mobile devices over the unlicensed spectrum. However, the interference from surrounding terminals is uncontrollable due to the opportunistic channel access. In this paper, we consider a cellular Internet of UAVs to guarantee the Quality-of-Service (QoS), where the sensory data can be transmitted to the mobile devices either by UAV-to-Device (U2D) communications over cellular networks, or directly through the base station (BS). Since UAVs’ sensing and transmission may influence their trajectories, we study the trajectory design problem for UAVs in consideration of their sensing and transmission. This is a Markov decision problem (MDP) with a large state-action space, and thus, we utilize multi-agent deep reinforcement learning (DRL) to approximate the state-action space, and then propose a multi-UAV trajectory design algorithm to solve this problem. Simulation results show that our proposed algorithm can achieve a higher total utility than policy gradient algorithm and single-agent algorithm. [ABSTRACT FROM AUTHOR]

In recent years, how to use renewable energy to reduce the energy cost of internet data center (IDC) has been an urgent problem to be solved. More and more solutions are beginning to consider machine learning, but many of the existing methods need to take advantage of some future information, which is difficult to obtain in the actual operation process. In this paper, we focus on reducing the energy cost of IDC by controlling the energy flow of renewable energy without any future information. we propose an efficient energy dynamic control algorithm based on the theory of reinforcement learning, which approximates the optimal solution by learning the feedback of historical control decisions. For the purpose of avoiding overestimation, improving the convergence ability of the algorithm, we use the double Q -method to further optimize. The extensive experimental results show that our algorithm can on average save the energy cost by 18.3% and reduce the rate of grid intervention by 26.2% compared with other algorithms, and thus has good application prospects. [ABSTRACT FROM AUTHOR]

Multi-agent reinforcement learning (MARL) has been successfully applied in many fields. In MARL, the policy evaluation problem is one of crucial problems. In order to solve this problem, distributed Temporal-Difference (TD) learning algorithm is one of the most popular methods in a cooperative manner. Despite its empirical success, however, the theory of the adaptive variant of distributed TD learning still remain limited. To fill this gap, we propose an adaptive distributed temporal-difference algorithm (referred to as MS - ADTD) under Markovian sampling over time-varying networks. Furthermore, we rigorously analyze the convergence of MS - ADTD , the theoretical results show that the local estimation can converge linearly to the optimal neighborhood. Meanwhile, the theoretical results are verified by simulation experiments. • This paper proposes a distributed adaptive TD learning algorithm under Markovian sampling. • It theoretically analyzes the non-asymptotic convergence performance of the proposed algorithm. • It verifies the performance of the proposed algorithm by experiments. [ABSTRACT FROM AUTHOR]

• Peoples who're specially affected with heart sickness and it's far one of the man-kill illnesses in the international level. • Most of researchers to awareness at the prediction, clustering, rule generation, decision tree and machine learning algorithm for figuring out and predicting the danger of the sufferers primarily based on the medical information. • The researcher can pick out the time and reminiscence wanted for the execution. • There are many different types of machine learning algorithms are categorized into three important classifications namely unsupervised learning, supervised learning and reinforcement learning. • This research paper includes the six elements to evaluate the overall performance of K-Means, Navie-Bayes and enhanced PSNB-IMR Algorithm with various parameters. Peoples who're specially affected with heart sickness and it's far one of the man-kill illnesses in the international level. Most of researchers to awareness at the prediction, clustering, rule generation, decision tree and machine learning algorithm for figuring out and predicting the danger of the sufferers primarily based on the medical information. The overall performances of the crucial functions are based on the machine-learning concept. By studying the algorithm, the researcher can pick out the time and reminiscence wanted for the execution. As such, there are many different types of machine learning algorithms are categorized into three important classifications namely unsupervised learning, supervised learning and reinforcement learning. Unsupervised learning consists of all varieties of clustering algorithms at the same time as supervised learning algorithm consists of all of the category strategies. But the author is considered the two algorithms are Supervised and Unsupervised learning algorithm to examine the overall performance. This research paper includes the six elements to evaluate the overall performance of K-Means, Navie-Bayes and enhanced PSNB-IMR Algorithm with various parameters. [ABSTRACT FROM AUTHOR]

In reinforcement learning (RL), the exploration/exploitation (E/E) dilemma is a very crucial issue, which can be described as searching between the exploration of the environment to find more profitable actions, and the exploitation of the best empirical actions for the current state. We focus on the single trajectory RL problem where an agent is interacting with a partially unknown MDP over single trajectories, and try to deal with the E/E in this setting. Given the reward function, we try to find a good E/E strategy to address the MDPs under some MDP distribution. This is achieved by selecting the best strategy in mean over a potential MDP distribution from a large set of candidate strategies, which is done by exploiting single trajectories drawn from plenty of MDPs. In this paper, we mainly make the following contributions: (1) We discuss the strategy-selector algorithm based on formula set and polynomial function. (2) We provide the theoretical and experimental regret analysis of the learned strategy under an given MDP distribution. (3) We compare these methods with the 'state-of-the-art' Bayesian RL method experimentally. [ABSTRACT FROM AUTHOR]

Deep Reinforcement Learning systems are now a hot topic in Machine Learning for their effectiveness in many complex tasks, but their application in safety-critical domains (e.g., robot control or self-autonomous driving) remains dangerous without mechanism to detect and prevent risk situations. In Deep RL, such risk is mostly in the form of adversarial attacks, which introduce small perturbations to sensor inputs with the aim of changing the network-based decisions and thus cause catastrophic situations. In the light of these dangers, a promising line of research is that of providing these Deep RL algorithms with suitable defenses, especially when deploying in real environments. This paper suggests that this line of research could be greatly improved by the concepts from the existing research field of Safe Reinforcement Learning, which has been postulated as a family of RL algorithms capable of providing defenses against many forms of risks. However, the connections between Safe RL and the design of defenses against adversarial attacks in Deep RL remain largely unexplored. This paper seeks to explore precisely some of these connections. In particular, this paper proposes to reuse some of the concepts from existing Safe RL algorithms to create a novel and effective instance-based defense for the deployment stage of Deep RL policies. The proposed algorithm uses a risk function based on how far a state is from the state space known by the agent, that allows identifying and preventing adversarial situations. The success of the proposed defense has been evaluated in 4 Atari games. [ABSTRACT FROM AUTHOR]

Graphical avatars have gained popularity in many application domains such as three-dimensional (3D) animation movies and animated simulations for product design. However, the methods to edit avatars' behaviors in the 3D graphical environment remained to be a challenging research topic. Since the hand-crafted methods are time-consuming and inefficient, the automatic actions of the avatars are required. To achieve the autonomous behaviors of the avatars, artificial intelligence should be used in this research area. In this paper, we present a novel approach to construct a system of automatic avatars in the 3D graphical environments based on the machine learning techniques. Specific framework is created for controlling the behaviors of avatars, such as classifying the difference among the environments and using hierarchical structure to describe these actions. Because of the requirement of simulating the interactions between avatars and environments after the classification of the environment, Reinforcement Learning is used to compute the policy to control the avatar intelligently in the 3D environment for the solution of the problem of different situations. Thus, our approach has solved problems such as where the levels of the missions will be defined and how the learning algorithm will be used to control the avatars. In this paper, our method to achieve these goals will be presented. The main contributions of this paper are presenting a hierarchical structure to control avatars automatically, developing a method for avatars to recognize environment and presenting an approach for making the policy of avatars' actions intelligently. [ABSTRACT FROM AUTHOR]

The key approaches for machine learning, particu- larly learning in unknown probabilistic environments, are new representations and computation mechanisms. In this paper, a novel quantum reinforcement learning (QRL) method is proposed by combining quantum theory and reinforcement learning (RL). Inspired by the state superposition principle and quantum paral- lelism, a framework of a value-updating algorithm is introduced. The state (action) in traditional RL is identified as the eigen state (eigen action) in QRL. The state (action) set can be represented with a quantum superposition state, and the eigen state (eigen action) can be obtained by randomly observing the simulated quantum state according to the collapse postulate of quantum measurement. The probability of the eigen action is determined by the probability amplitude, which is updated in parallel ac- cording to rewards. Some related characteristics of QRL such as convergence, optimality, and balancing between exploration and exploitation are also analyzed, which shows that this approach makes a good tradeoff between exploration and exploitation using the probability amplitude and can speedup learning through the quantum parallelism. To evaluate the performance and practi- cability of QRL, several simulated experiments are given, and the results demonstrate the effectiveness and superiority of the QRL algorithm for some complex problems. This paper is also an effective exploration on the application of quantum computation to artificial intelligence. [ABSTRACT FROM AUTHOR]

Multi-tier storage systems are becoming more and more widespread in the industry. They have more tunable parameters and built-in policies than traditional storage systems, and an adequate configuration of these parameters and policies is crucial for achieving high performance. A very important performance indicator for such systems is the response time of the file I/O requests. The response time can be minimized if the most frequently accessed (“hot”) files are located in the fastest storage tiers. Unfortunately, it is impossible to know a priori which files are going to be hot, especially because the file access patterns change over time. This paper presents a policy-based framework for dynamically deciding which files need to be upgraded and which files need to be downgraded based on their recent access pattern and on the system’s current state. The paper also presents a reinforcement learning (RL) algorithm for automatically tuning the file migration policies in order to minimize the average request response time. A multi-tier storage system simulator was used to evaluate the migration policies tuned by RL, and such policies were shown to achieve a significant performance improvement over the best hand-crafted policies found for this domain. [ABSTRACT FROM AUTHOR]

Abstract: Reinforcement learning (RL) has received some attention in recent years from agent-based researchers because it deals with the problem of how an autonomous agent can learn to select proper actions for achieving its goals through interacting with its environment. Although there have been several successful examples demonstrating the usefulness of RL, its application to manufacturing systems has not been fully explored yet. In this paper, Q-learning, a popular RL algorithm, is applied to a single machine dispatching rule selection problem. This paper investigates the application potential of Q-learning, a widely used RL algorithm to a dispatching rule selection problem on a single machine to determine if it can be used to enable a single machine agent to learn commonly accepted dispatching rules for three example cases in which the best dispatching rules have been previously defined. This study provided encouraging results that show the potential of RL for application to agent-based production scheduling. [Copyright &y& Elsevier]