Search

The lighting requirements are subjective and one light setting cannot work for all. However, there is little work on developing smart lighting algorithms that can adapt to user preferences. To address this gap, this paper uses fuzzy logic and reinforcement learning to develop an adaptive lighting algorithm. In particular, we develop a baseline fuzzy inference system (FIS) using the domain knowledge, generating light recommendation based on a set of intuitive rules. These rules, derived from existing literature, are based on environmental conditions i.e. daily glare index, and user information including age, activity, and chronotype. Through a feedback mechanism, the user interacts with the algorithm, correcting the algorithm output to their preferences. We interpret these corrections as rewards to a Q-learning algorithm, which tunes the FIS parameters online to match the user preferences. The Q-learning is a model-free learning algorithm that learns to act optimally by interacting with the user and the rewards it receives. This allows the proposed algorithm to work in a model-free manner, effectively handling the uncertainties that might arise from the individualistic preferences of users. To the authors' best knowledge, this algorithm is pioneering work in designing intelligent algorithms for personalized lighting control, featuring several elements of novelty, including the number of environmental and user-related inputs, the continuous control of light intensity as opposed to common on/off control, and the ability to learn user preferences. The algorithm is implemented in a real aircraft cabin and is evaluated in an extensive user study. The implementation results demonstrate that the developed algorithm possesses the capability to learn user preferences while successfully adapting to a wide range of environmental conditions and user characteristics. This underscores its viability as a potent solution for intelligent light management, featuring advanced learning capabilities. • Intelligent lighting algorithm with the ability to learn from and adapt to user preferences. • A fuzzy inference system tunes lighting based on environment and user traits. • The developed algorithm is tested and verified via an in-depth user study. [ABSTRACT FROM AUTHOR]

Grid-interactive building control poses a critical challenge in the context of grid modernization and decarbonization. Recently, various artificial intelligence-based optimal control approaches have been proposed, providing innovative solutions to optimal building control problems. However, researchers and practitioners are now confronted with a dilemma when selecting appropriate control strategies, ranging from model-based (knowledge-incorporating) to learning-based (data-driven) approaches, and hybrid methods combining them. Although each algorithm in existing literature claims superiority over specific baselines, their performance has never been systematically compared and analyzed, owing to the absence of a unified platform for comprehensive evaluation. To fill this knowledge gap, we identify and implement all state-of-the-art approaches within a modular training and evaluation framework, assessing their efficacy in a grid-interactive building control problem. In this paper, we also introduce a streamlined hybrid method that complements existing hybrid approaches. Our comparative study reveals and quantifies the advantages of hybrid methods: on average, they achieve near-optimal control while requiring less than 14% of the online computation of traditional model-based methods. To achieve this performance, they need as few as 2% of the training samples of purely learning-based methods. Finally, we provide insights into the merits, limitations, and implementation of each method to help researchers better understand the state of the art and future directions. The software framework implemented in this study is open-sourced to facilitate future research. [ABSTRACT FROM AUTHOR]

The intermittent integration of renewable energy sources and the enhancement of energy-saving awareness on the demand side have posed significant challenges to energy management in smart grids. To address the supply–demand imbalance issues caused by these challenges, this paper proposes a novel multiple time-scale dispatch model. The model utilizes demand response strategy for demand side management (DSM) and employs dynamic pricing strategy for supply-side management. Furthermore, an advanced approach based on federated learning and Twin Delayed Deep Deterministic(TD3) policy gradient algorithm is proposed to construct a federated learning mechanism among multi agents. This approach performs distributed optimization on the multi-agent system to derive the optimal network weights for multiple stakeholders, resulting in the generation of an optimal dynamic pricing strategy that ensures the economic efficiency and secure operation of the smart grid. From the methodological perspective, this approach enhances privacy protection and decision-making autonomy for each stakeholder, while improving the optimization efficiency of the reinforcement learning (RL) algorithm. Based on simulation results, it is evident that the proposed method facilitates economically efficient transactions between supply and demand, and the improved distributed multi-agent federated reinforcement learning (MAFRL) approach holds promise as a feasible and optimal optimization method for economic dispatch in smart grid. [ABSTRACT FROM AUTHOR]

Online ride-hailing has become one of the most important transportation ways. In the ride-hailing system, how to efficiently match orders with vehicles and dispatch idle vehicles are key issues. The ride-hailing platform needs to match orders with vehicles and dispatch idle vehicles efficiently to maximize social welfare. However, the matching and dispatching decisions at the current round may affect the supply and demand of ride-hailing in the future rounds since they will affect the future vehicle distributions in different geographical zones. In fact, vehicles in different zones at different times may have different values for the matching and dispatching results. In this paper, we use the vehicle value function to characterize the spatio-temporal value of vehicles in each zone and then use it to design the order matching and idle vehicle dispatching algorithm to improve the long-term social welfare. In addition, in the order matching, passengers may untruthfully report the maximum price they are willing to pay to maximize their own profits, which can affect the order matching and thus may result in the losses of the long-term social welfare. Therefore, we design a VCG based pricing algorithm to prevent the strategic behavior of passengers. We further run experiments to evaluate the proposed algorithm. The experimental results show that our algorithm can outperform the state-of-the-art algorithm in terms of social welfare by 11.7% and service ratio by 11.1%. This work can provide some useful insights for the online ride-hailing platform to design practical order matching and pricing strategies. [Display omitted] • We design a vehicle value function to optimize order matching and improve the long-term social welfare by using a reinforcement learning method. • We consider the dispatching of idle vehicles to a zone as a virtual order and process idle vehicle dispatching issue as (virtual) order matching as well. • We design a VCG based pricing method to prevent the strategic behavior of passengers and ensure positive profits for the platform. [ABSTRACT FROM AUTHOR]

Nowadays, economic and environmental concerns in production have become increasingly significant. To address these issues, the Combined Economic and Emission Dispatch (CEED) problem has been introduced to optimize the power generation process by considering fuel cost and emitted substances. However, due to the nonlinearity and nonconvexity of the objective function, the optimization of CEED remains a challenge. In this paper, we develop a Reinforcement Learning-based Adaptive Differential Evolution (RLADE) algorithm to enhance the optimization performance. The mutation strategy and crossover probability of RLADE are optimized using Reinforcement Learning (RL) to respectively ensure better convergence speed and searchability. Additionally, two modifications of RL, namely the adaptive population size-based state division and fitness-ranking-based reward mechanism, are proposed to improve the accuracy of state division and reward calculation in RL. The experiments conducted in this paper consider two objective formulation methods of CEED problems, namely the quadratic and cubic criterion functions. The mean values and standard deviations of the obtained solutions were utilized to assess the performance of RLADE, as well as other comparative algorithms, namely DE algorithm and two RL-based DE variants. The results clearly demonstrate that RLADE surpasses its counterparts with proportion of 100%, 85.7%, and 100% for the 6-unit and 11-unit quadratic CEED problems, as well as cubic criterion functions, in terms of both search accuracy and convergence ability. Furthermore, the significance of RLADE's superiority is confirmed through the Wilcoxon's signed rank test. [ABSTRACT FROM AUTHOR]

Aeroengine fault detection is an important means to ensure flight safety. The application premise of data driven fault detection method is that all data come from the same distribution. However, this assumption is invalid in the actual engine fault detection, because the engine state will change with the increase of operating time, and the collected data will also have distribution differences. Due to the high cost of collecting engine data, it is difficult to collect enough data from the current state. Fortunately, transfer learning can transfer data information from other fields to the target field, thus alleviating the problem of data scarcity in the target domain. Therefore, from the idea of transfer learning, this paper proposes a cross domain aeroengine fault detection method, viz. transfer learning based on kernel perception algorithm with uneven margins (TL-KPAUM). The proposed method is divided into two stages. In the first stage, KPAUM is trained using the data in the source domain to extract the information in the source domain. In the second stage, the data in the target domain is used to realize the adaptation to the target domain. Compared with several baseline methods, TL-KPAUM has better performance when the amount of data in the target domain is small. Finally, using the simulation data of turbofan engine, the fault detection experiments of degraded state are designed, and the results show that this method is effective. [Display omitted] • A transfer learning algorithm is proposed for aeroengine fault detection. • The proposed method is applied to engine fault detection in different domains. • Simulation results show the effectiveness of this method compared with other methods. • The proposed method has good performance in the case of sparse data in the target domain. [ABSTRACT FROM AUTHOR]

The collaborative optimization dispatching of multiple energy flows plays a crucial role in achieving the economic and efficient low-carbon operation of integrated energy systems (IESs). However, the dispatching problem for IESs is characterized by high dimensionality, non-linearity, and complex coupling. Furthermore, the integration of renewable energy sources and flexible loads has transformed the IES into a complex dynamic system with significant uncertainty. Traditional intelligent optimization algorithms exhibit poor adaptability and lengthy solution computation time when tackling such problems. In contrast, deep reinforcement learning (DRL), as an interactive trial-and-error learning method, has shown improved decision-making capabilities. In view of this, a data-driven soft actor-critic (SAC) deep reinforcement learning-based approach is proposed in this paper for interval optimal dispatch of IESs considering multiple uncertainties. First, the basic principle of SAC reinforcement learning is introduced in detail, and the basic framework of reinforcement learning for interval optimal scheduling of IESs is constructed. Then, the environment model of agent interaction is constructed, and the action and state space of DRL, as well as the reward mechanism and neural network structure, are designed. Finally, a typical IES case is experimentally analyzed and compared with three popular DRL algorithms and five state-of-the-art intelligent optimization algorithms. The experimental results demonstrate the advantages and effectiveness of the proposed method in solving the optimal dispatching of IESs. [ABSTRACT FROM AUTHOR]

We explore the reinforcement learning approach to designing controllers by extensively discussing the case of a quadcopter attitude controller. We provide all details allowing to reproduce our approach, starting with a model of the dynamics of a crazyflie 2.0 under various nominal and non-nominal conditions, including partial motor failures and wind gusts. We develop a robust form of a signal temporal logic to quantitatively evaluate the vehicle's behavior and measure the performance of controllers. The paper thoroughly describes the choices in training algorithms, neural net architecture, hyperparameters, observation space in view of the different performance metrics we have introduced. We discuss the robustness of the obtained controllers, both to partial loss of power for one rotor and to wind gusts and finish by drawing conclusions on practical controller design by reinforcement learning. [ABSTRACT FROM AUTHOR]

This paper presents a comprehensive literature review on Reinforcement Learning (RL) techniques used in Intrusion Detection Systems (IDS), Intrusion Prevention Systems (IPS), Internet of Things (IoT) and Identity and Access Management (IAM). This study reviews scientific documents such as journals and articles, from 2010 to 2021, extracted from the Science Direct, ACM, IEEEXplore, and Springer database. Most of the research articles published in 2020 and 2021, for cybersecurity and RL are for IDS classifiers and resource optimization in IoTs. Some datasets used for training RL-based IDS algorithms are NSL-KDD, CICIDS, and AWID. There are few datasets and publications for IAM. The few that exist focus on the physical layer authentication. The current state of the art lacks standard evaluation criteria, however, we have identified parameters like detection rate, precision, and accuracy which can be used to compare the algorithms employing RL. This paper is suitable for new researchers, students, and beginners in the field of RL who want to learn about the field and identify problem areas. [ABSTRACT FROM AUTHOR]

Conventional microtask crowdsourcing platforms rely on a random task distribution strategy and repeatedly assign tasks to workers. This strategy known as repeated labelling suffers from two shortcomings of high cost and low accuracy as a result of making random distributions. To overcome such shortcomings researchers have introduced task assignment as a substitute strategy. In this strategy, an algorithm selectively chooses suitable tasks for an online worker. Hence, task assignment has gained attentions from researchers to reduce the cost of microtasking whiling increasing its accuracy. However, the existing algorithms on task assignment suffer from four shortcomings as: (i) human intervention, (ii) reliance on a rough estimation of ground truth, (iii) reliance on workers’ dynamic capabilities and (iv) lack of ability in dealing with sparsity. To overcome these shortcomings this paper proposes a new task assignment algorithm known as LEarning Automata based Task assignment (LEATask), that works based on the similarities of workers in performance. This algorithm has two stages of exploration and exploitation. In exploration stage, first a number of workers are hired to learn their reliability. Then, LEATask clusters the hired workers using a given clustering algorithm, and for each cluster generates learning automata. Later, the clusters of workers along with their attached learning automata will be used in exploitation stage. Exploitation stage initially assigns a number of tasks to a newly arrived worker to learn the worker’s reliability. Then, LEATask identifies the cluster of worker. Based on the cluster that worker resides in and the attached learning automata, the next tasks will be assigned to the new worker. LEATask has been empirically evaluated using several real datasets and compared against the baseline and novel algorithms, in terms of root mean square error. The comparisons indicates LEATask consistently is showing better or comparable performance. [ABSTRACT FROM AUTHOR]

This paper evaluates adaptive Q-learning (AQL) and single-partition adaptive Q-learning (SPAQL), two algorithms for efficient model-free episodic reinforcement learning (RL), in two classical control problems (Pendulum and CartPole). AQL adaptively partitions the state–action space of a Markov decision process (MDP), while learning the control policy, i.e. , the mapping from states to actions. The main difference between AQL and SPAQL is that the latter learns time-invariant policies, where the mapping from states to actions does not depend explicitly on the time step. This paper also proposes the SPAQL with terminal state (SPAQL-TS), an improved version of SPAQL tailored for the design of regulators for control problems. The time-invariant policies are shown to result in a better performance than the time-variant ones in both problems studied. These algorithms are particularly fitted to RL problems where the action space is finite, as is the case with the CartPole problem. SPAQL-TS solves the OpenAI GymCartPole problem, while also displaying a higher sample efficiency than trust region policy optimization (TRPO), a standard RL algorithm for solving control tasks. Moreover, the policies learned by SPAQL are interpretable, while TRPO policies are typically encoded as neural networks, and therefore hard to interpret. Yielding interpretable policies while being sample-efficient are the major advantages of SPAQL. The code for the experiments is available at https://github.com/jaraujo98/SinglePartitionAdaptiveQLearning. • Two recent Q-learning algorithms, AQL and SPAQL, are evaluated on two classical control benchmarks. • Based on insights from control theory, a new algorithm, SPAQL-TS, is introduced. • It is shown that both SPAQL and SPAQL-TS outperform TRPO in the Cartpole problem. [ABSTRACT FROM AUTHOR]

Deep Reinforcement Learning systems are now a hot topic in Machine Learning for their effectiveness in many complex tasks, but their application in safety-critical domains (e.g., robot control or self-autonomous driving) remains dangerous without mechanism to detect and prevent risk situations. In Deep RL, such risk is mostly in the form of adversarial attacks, which introduce small perturbations to sensor inputs with the aim of changing the network-based decisions and thus cause catastrophic situations. In the light of these dangers, a promising line of research is that of providing these Deep RL algorithms with suitable defenses, especially when deploying in real environments. This paper suggests that this line of research could be greatly improved by the concepts from the existing research field of Safe Reinforcement Learning, which has been postulated as a family of RL algorithms capable of providing defenses against many forms of risks. However, the connections between Safe RL and the design of defenses against adversarial attacks in Deep RL remain largely unexplored. This paper seeks to explore precisely some of these connections. In particular, this paper proposes to reuse some of the concepts from existing Safe RL algorithms to create a novel and effective instance-based defense for the deployment stage of Deep RL policies. The proposed algorithm uses a risk function based on how far a state is from the state space known by the agent, that allows identifying and preventing adversarial situations. The success of the proposed defense has been evaluated in 4 Atari games. [ABSTRACT FROM AUTHOR]

Contemporary malware makes wide use of techniques to evade popular detection approaches. Behavior-based detection is the most powerful approach to malware detection. This approach is based on system call sequences to model a malicious behavior. A recently immersed malware to defeat behavior-based detection approach is Multi-process malware. This malware is the consequence of multiple processes cooperating to fulfill a malicious task each of which performing a partition of main task and none of them shows an identifiable malicious behavior. In this paper, we have presented a new method called PbMMD for detecting Multi-process malware. In this method, we attempt to inspect the whole processes running on the system and discover collaborative processes by finding processes running along a common execution policy. Beforehand we have learned different execution policy by employing reinforcement algorithm. Finally we decide against a Multi-process malicious behavior by analyzing the cumulative behavior of identified collaborative processes. [ABSTRACT FROM AUTHOR]

Abstract: This paper introduces a driving danger-level prediction system that uses multiple sensor inputs and statistical modeling to predict the driving risk. Three types of features were collected for the research, specifically the vehicle dynamic parameter, the driver''s physiological data and the driver''s behavior feature. To model the temporal patterns that lead to safe/dangerous driving state, several sequential supervised learning algorithms were evaluated in the paper, including hidden Markov model, conditional random field and reinforcement learning. Experimental results showed that using reinforcement learning based method with the vehicle dynamic parameters feature outperforms the rest algorithms, and adding the other two features could further improve the prediction accuracy. Based on the result, a live driving danger-level prediction prototype system was developed. Compared to many previous researches that focused on monitoring the driver''s vigilance level to infer the possibility of potential driving risk, our live system is non-intrusive to the driver, and hence it is very desirable for driving danger prevention applications. Subjective on-line user study of our prototype system gave promising results. [Copyright &y& Elsevier]

Abstract: Reinforcement learning (RL) has received some attention in recent years from agent-based researchers because it deals with the problem of how an autonomous agent can learn to select proper actions for achieving its goals through interacting with its environment. Although there have been several successful examples demonstrating the usefulness of RL, its application to manufacturing systems has not been fully explored yet. In this paper, Q-learning, a popular RL algorithm, is applied to a single machine dispatching rule selection problem. This paper investigates the application potential of Q-learning, a widely used RL algorithm to a dispatching rule selection problem on a single machine to determine if it can be used to enable a single machine agent to learn commonly accepted dispatching rules for three example cases in which the best dispatching rules have been previously defined. This study provided encouraging results that show the potential of RL for application to agent-based production scheduling. [Copyright &y& Elsevier]

Abstract: Reinforcement learning (RL) has been applied to many fields and applications, but there are still some dilemmas between exploration and exploitation strategy for action selection policy. The well-known areas of reinforcement learning are the Q-learning and the Sarsa algorithms, but they possess different characteristics. Generally speaking, the Sarsa algorithm has faster convergence characteristics, while the Q-learning algorithm has a better final performance. However, Sarsa algorithm is easily stuck in the local minimum and Q-learning needs longer time to learn. Most literatures investigated the action selection policy. Instead of studying an action selection strategy, this paper focuses on how to combine Q-learning with the Sarsa algorithm, and presents a new method, called backward Q-learning, which can be implemented in the Sarsa algorithm and Q-learning. The backward Q-learning algorithm directly tunes the Q-values, and then the Q-values will indirectly affect the action selection policy. Therefore, the proposed RL algorithms can enhance learning speed and improve final performance. Finally, three experimental results including cliff walk, mountain car, and cart–pole balancing control system are utilized to verify the feasibility and effectiveness of the proposed scheme. All the simulations illustrate that the backward Q-learning based RL algorithm outperforms the well-known Q-learning and the Sarsa algorithm. [Copyright &y& Elsevier]

The Deep Reinforcement Learning (DRL) algorithm, Proximal Policy Optimization (PPO2), is deployed on a custom spacecraft (S/C) build and loss model to determine if an Artificial Intelligence (AI) can learn to monitor satellite constellation health and determine an optimal replacement strategy. A custom environment is created to simulate how S/C are built, launched, generate revenue, and finally decay. The reinforcement learning agent successfully learned an optimal policy for two models: a Simplified Model where the financial cost of actions is ignored; and an Advanced Model where the financial cost of actions is a major element. In both models the AI monitors the constellations and takes multiple strategic and tactical actions to replace satellites to maintain constellation performance. The Simplified Model showed that the PPO2 algorithm was able to converge on an optimal solution after ∼ 200,000 simulations. The Advanced Model was much more difficult for the AI to learn, and thus, the performance drops during the early episodes, but eventually converges to an optimal policy at ∼ 25,000,000 simulations. With the Advanced Model, the AI is taking actions that are successfully providing strategies for constellation management and satellite replacements which include these actions' financial implications. Thus, the methods in this paper provide initial research developments towards a real-world tool and an AI application that can aid various Aerospace businesses in managing Low Earth Orbit (LEO) constellations. This type of AI application may become imperative for deploying and maintaining small satellite mega-constellations. [ABSTRACT FROM AUTHOR]

Hyperparameter optimization is critical for the performance of machine learning algorithms. Significant efforts have been dedicated to improve the final accuracy of algorithm by hyperparameter tuning. However, some indicators (such as latency, cpu utilization) are also very important in the actual environment. In this paper, we propose a novel method EMORL (E ffective M ulti- O bjective R einforcement L earning) based on multi-objective reinforcement learning for hyperparameter optimization to solve the above limitations. Specifically, we extend hyperparameter optimization problem to the reinforcement learning framework and employ an agent to select hyperparameters sequentially, and design a scalarization function that combines accuracy and latency as a multi-objective reward to guide the policy update. To improve the efficiency of hyperparameter optimization, previously successful configuration is reused for reshaping the advantage function. In the experiment, we apply the proposed method to tune the hyperparameters of the eXtreme Gradient Boosting on 101 tasks and convolutional neural networks on 2 tasks. The experimental results demonstrate that the proposed method is better than other methods in most tasks, especially in terms of latency. In addition, we verify the various components of the proposed method through ablation experiments. [ABSTRACT FROM AUTHOR]

Continuous control has attracted enormous attention due to its essential role in real-world applications. However, it is considerably difficult to be addressed through explicitly modeling in practice. As promising approaches, model-free policy gradient (PG) based methods in reinforcement learning (RL), however, suffer from slow convergence and complex computation owing to the high variance of gradient estimating and sophisticated backpropagation. Therefore, in this paper, a gradient-free policy gradient algorithm with PSO-based parameter exploration (PG-PSOPE) is proposed for continuous control tasks. To reduce variance and improve convergence rate, the PSO is combined with PG to provide a novel way for training policy network in RL. Experimental results of simulated physical control tasks verify the effectiveness of the proposed algorithm. Besides, the PG-PSOPE is superior in both convergence speed and final performance to the typical on-policy PG and the off-policy deep RL method. Furthermore, the PG-PSOPE exhibits the simplicity and high effectiveness by comparison of training time under different tasks, and its running time is reduced by 58 times compared with other gradient-based methods for the best case. [ABSTRACT FROM AUTHOR]