Search

With the rapid development of renewable energy and the increasing maturity of energy storage technology, microgrids are quickly becoming popular worldwide. The stochastic scheduling problem of microgrids can increase operational costs and resource wastage. In order to reduce operational costs and optimize resource utilization efficiency, the real-time scheduling of microgrids becomes particularly important. After collecting extensive data, reinforcement learning (RL) can provide good strategies. However, it cannot make quick and rational decisions in different environments. As a method with generalization ability, meta-learning can compensate for this deficiency. Therefore, this paper introduces a microgrid scheduling strategy based on RL and meta-learning. This method can quickly adapt to different environments with a small amount of training data, enabling rapid energy scheduling policy generation in the early stages of microgrid operation. This paper first establishes a microgrid model, including components such as energy storage, load, and distributed generation (DG). Then, we use a meta-reinforcement learning framework to train the initial scheduling strategy, considering the various operational constraints of the microgrid. The experimental results show that the MAML-based RL strategy has advantages in improving energy utilization and reducing operational costs in the early stages of microgrid operation. This research provides a new intelligent solution for microgrids' efficient, stable, and economical operation in their initial stages. [ABSTRACT FROM AUTHOR]

This paper proposes a novel cloud computingbased parallel deep reinforcement learning (DRL) energy management strategy (EMS) for connected plug-in hybrid vehicles. First, a proximal policy optimization (PPO) algorithm is developed. Since the cloud computing can reduce the computational burden of the connected vehicles, the PPO is deployed in the proposed cloud computing-based EMS. In order to improve the strategy adaptation, a parallel mechanism is proposed to achieve the information interaction with multiple vehicles. Considering the real-time control requirements, thread pool is proposed and applied in the cloud computing based parallel EMS. The thread pool-based strategy provides an efficient real-time control ability and strategy improvement solution. To verify the PPO based EMS, the dynamic programming, deep Q-network and double deep Q-network strategies are developed for comparison. It is found that the PPO can achieve similar fuel efficiency improvement with the DP strategy among the three DRL algorithms. For parallel training of multiple connected vehicles, the cloud computing-based parallel EMS improves fuel economy by approximately 7.7%. Threadpool based parallel real-time EMS reduces average time for computational interactions by 20% and further improves the fuel efficiency. The proposed strategy has the advantages of realtime control, adaptability and continuous learning for improved fuel efficiency. [ABSTRACT FROM AUTHOR]

The Smart Grid AMI is at the forefront of modernizing energy management for efficient energy distribution and maintaining grid reliability. However, the seamless flow of data within AMI introduces critical security challenges, necessitating innovative approaches to ensure secure data transmission. This paper addresses these challenges by proposing a Quantum Key Distribution (QKD) based on Multi-Objective Optimization (MOO) and Reinforcement Learning (RL) framework for secure data transmission within the Smart Grid AMI. Primarily, we formulate the routing problem as a MOO challenge with four primary objectives namely, energy efficiency, latency minimization, reliability, and security to capture the essence of efficient and secure data transmission. Further, we introduce an RL agent based on the Proximal Policy Optimization (PPO) algorithm for robust policy learning. The RL agent by exploring diverse routing actions and receiving rewards optimizes the routing decisions within the network environment based on the objectives specified in the multi-objective function (MOF). A novel MOF quantifies trade-offs between security and performance metrics, integrating QKD-based security metrics with traditional optimization objectives. This function guides the RL agent in making informed routing decisions. Through extensive simulations using the NS-2 simulator in a realistic Smart Grid AMI environment, the proposed approach obtained energy consumption of 750 J, latency of 55 ms, and security level of 96% and also revealed significant improvements compared to conventional techniques widely employed for secured communication in AMI networks. Overall, the simulation results exhibited that the proposed method showed outstanding performance by achieving energy efficiency of 0.95, latency reduction of 0.92, reliability improvement of 0.94, and security enhancement of 0.96. Overall, this integration offers a pioneering solution to address the evolving security challenges in the Smart Grid landscape and contributes to the advancement of secure data transmission in Smart Grid AMI networks, fostering a resilient and secure energy ecosystem. [ABSTRACT FROM AUTHOR]

The service life and fuel consumption of fuel cell system (FCS) are the main factors limiting the commercialization of fuel cell electric vehicles (FCEV). Effective energy management strategies (EMS) can reduce fuel consumption during the cycle and prolong the service life of FCS. This paper proposes an energy management strategy based on the deep reinforcement learning (DRL) algorithm, deep Q-learning (DQL). Considering the unstable performance of conventional DQL during the training process, a new algorithm called Double Deep Q Learning (DDQL) is introduced. The DDQL uses a target evaluation network to evaluate output actions and a delayed update strategy to improve the convergence and stability of DRL. This article trains the strategy using UDDS cycle, tests it using combined cycles UDDS-WLTC-NEDC, and compares it with traditional ECM-based EMS. The results demonstrate that under the combined cycle, the strategy effectively reduced FCS voltage degradation by 50%, maintained fuel economy, and ensured consistency between the initial and final state of charge (SOC) of LIB. [ABSTRACT FROM AUTHOR]

This paper provides a comprehensive review of the integration of advanced power management systems and learning techniques in the field of robotics. It identifies the critical roles these areas play in reshaping the capabilities of robotic systems across diverse applications. To begin, it highlights the significance of efficient power usage in modern robotics. The paper explains how advanced power converters effectively control voltage, manage current and shape waveforms, thereby optimizing energy utilization. These converters ensure that robotic components receive the precise voltage levels they require, leading to improved motor performance and enabling precise control over motor behavior. Consequently, this results in extended operational times and increased design flexibility. Furthermore, the review explores the integration of learning approaches, emphasizing their substantial impact on robotic perception, decision-making and autonomy. It discusses the application of techniques such as reinforcement learning, supervised learning and unsupervised learning, showcasing their applications in areas like object recognition, semantic segmentation, sensor fusion and anomaly detection. By utilizing these learning methods, robots become more intelligent, adaptable and capable of autonomous operation across various domains. By examining the interaction between advanced power management and learning integration, this review anticipates a future where robots operate with increased efficiency, adapt to various tasks and drive technological innovation across a wide range of industries. [ABSTRACT FROM AUTHOR]

• This study proposes an hour-ahead demand response (DR) for PV-based smart homes utilizing the AI method in digital twin while taking into account customer energy consumption and discomfort; • This study adopts a MARL approach for ensuring optimum and decentralized decision-making among several devices; • This study presents a steady cost prediction method utilizing ANN for overcoming the uncertainties in upcoming costs; • This study also compares the energy prices under 2 various scenarios including and excluding AI-enabled DR. Energy management (EM) systems need to have the flexibility to take the optimum real-time decision in the face of constantly varying market factors. Demand response (DR) has become the newest method of improving the performance and reliability of the electrical system. Here, an hour-ahead DR algorithm is proposed for EM at home. This paper presents an artificial neural network approach that uses stable cost predictions as a method for dealing with upcoming price uncertainties. For making optimum and decentralized decisions for various household devices, multi-agent reinforcement learning has been used along with predicted upcoming costs. This paper conducts the simulation with shiftable, non-shiftable, and controllable loads to determine the effectiveness of this suggested EM strategy. Based on the outcomes of the experiments, this suggested DR algorithm has capable of handling EM for a number of devices, minimizing consumer electricity expenses and discomfort prices, and helping the consumer considerably reduce its energy expenses in comparison to a benchmark using no DR. [ABSTRACT FROM AUTHOR]

Over the past few years, the Bitcoin-based financial trading system (BFTS) has created new challenges for the power system due to the high-risk consumption of mining devices. Briefly, such a problem would be a compelling incentive for cyber-attackers who intend to inflict significant infections on a power system. Simply put, an effort to phony up the consumption data of mining devices results in the furtherance of messing up the optimal energy management within the power system. Hence, this paper introduces a new cyber-attack named miner-misuse for power systems equipped by transaction tech. To overwhelm this dispute, this article also addresses an online coefficient anomaly detection approach with reliance on the reinforcement learning (RL) concept for the power system. On account of not being sufficiently aware of the system, we fulfilled the Observable Markov Decision Process (OMDP) idea in the RL mechanism in order to barricade the miner attack. The proposed method would be enhanced in an optimal and punctual way if the setting parameters were properly established in the learning procedure. So to speak, a hybrid mechanism of the optimization approach and learning structure will not only guarantee catching in the best and most far-sighted solution but also become the high converging time. To this end, this paper proposes an Intelligent Priority Selection (IPS) algorithm merging with the suggested RL method to become more punctual and optimum in the way of detecting miner attacks. Additionally, to conjure up the proposed detection approach's effectiveness, mathematical modeling of the energy consumption of the mining devices based on the hashing rate within BFTS is provided. The uncertain fluctuation related to the needed energy of miners makes energy management unpredictable and needs to be dealt with. Hence, the unscented transformation (UT) method can obtain a high chance of precisely modeling the uncertain parameters within the system. All in all, the F-score value and successful probability of attack inferred from results revealed that the proposed anomaly detection method has the ability to identify the miner attacks as real-time-short as possible compared to other approaches. [ABSTRACT FROM AUTHOR]

The hybrid electric train which operates without overhead wires or traditional power sources relies on hydrogen fuel cells and batteries for power. These fuel cell-based hybrid electric trains (FCHETs) are more efficient than those powered by diesel or electricity because they do not produce any tailpipe emissions making them an eco-friendly mode of transport. The target of this paper is to propose low-budget FCHETs that prioritize energy efficiency to reduce operating costs and minimize their impact on the environment. To this end, an energy management strategy [EMS] has been developed that optimizes the distribution of energy to reduce the amount of hydrogen required to power the train. The EMS achieves this by balancing battery charging and discharging. To enhance the performance of the EMS, proposes to use of a deep reinforcement learning (DRL) algorithm specifically the deep deterministic policy gradient (DDPG) combined with transfer learning (TL) which can improve the system's efficiency when driving cycles are changed. DRL-based strategies are commonly used in energy management and they suffer from unstable convergence, slow learning speed, and insufficient constraint capability. To address these limitations, an action masking technique to stop the DDPG-based approach from producing incorrect actions that go against the system's physical limits and prevent them from being generated is proposed. The DDPG+TL agent consumes up to 3.9% less energy than conventional rulebased EMS while maintaining the battery's charge level within a predetermined range. The results show that DDPG+TL can sustain battery charge at minimal hydrogen consumption with minimal training time for the agent. [ABSTRACT FROM AUTHOR]

With the advent of the Distributed Energy Resources within smart grid systems, traditional demand response management (DRM) models need to be redesigned to capture prosumers' energy consumption requests and dynamic behavior within the energy market. In this paper, a coalitional DRM model is introduced based on the principles of Game Theory and reinforcement learning to dynamically determine prosumers' formation in local energy trading communities and their optimal energy consumption. A hedonic game-theoretic model is introduced to enable prosumers to autonomously and dynamically select an energy trading community considering the partially available information regarding prosumers' energy generation and consumption characteristics and utility companies' provided rewards per community. Then, a log-linear reinforcement learning model is proposed to enable each prosumer to distributedly determine their optimal energy consumption. A detailed evaluation of the proposed coalitional DRM model is performed based on real data collected from the southwest area of the USA. [ABSTRACT FROM AUTHOR]

Due to the uncertainty of user's behaviour and other conditions, the design of energy efficiency improvement methods in buildings is challenging. In this paper, a building energy management method based on deep reinforcement learning is proposed, which solves the energy scheduling problem of buildings with renewable sources and energy storage system and minimises electricity costs while maintaining the user's comfort. Different from model-based methods, the proposed DRL agent makes decisions only by observing the measurable information without considering the dynamic of the building environment. Simulations based on real data verify the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

In the current era, the skyrocketing demand for energy necessitates a powerful mechanism to mitigate the supply–demand gap in intelligent energy infrastructure, i.e., the smart grid. To handle this issue, an intelligent and secure energy management system (EMS) could benefit end-consumers participating in the Demand–Response (DR) program. Therefore, in this paper, we proposed a real-time and secure incentive-based EMS for smart grid, i.e., RI-EMS approach using Reinforcement Learning (RL) and blockchain technology. In the RI-EMS approach, we proposed a novel reward mechanism for better convergence of the RL-based model using a Q-learning approach based on the greedy policy that guides the RL-agent for faster convergence. Then, the proposed RI-EMS approach designed a real-time incentive mechanism to minimize energy consumption in peak hours and reduce end-consumers' energy bills to provide incentives to the end-consumers. Experimental results show that the proposed RI-EMS approach induces end-consumer participation and increases customer profitabilities compared to existing approaches considering the different performance evaluation metrics such as energy consumption for end-consumers, energy consumption reduction, and total cost comparison to end-consumers. Furthermore, blockchain-based results are simulated and analyzed with the help of deployed smart contracts in a Remix Integrated Development Environment (IDE) with the parameters such as transaction efficiency and data storage cost. [ABSTRACT FROM AUTHOR]

The drawbacks of the "take-or-pay" conception led to the search for alternatives to mitigate its effects. This paper proposes an intelligent multistage control system consisting of two-phased to control the energy sources' operation in a connected-mode microgrid consisting of renewable energy sources and a diesel generator. The first phase is the forecasting process to predict the day's required fuel amount, thus dispensing with the take-or-pay method that forces the purchase of redundant quantities of fuel periodically. This stage depends on the deep neural network long short-term memory method. The second phase is the energy management to find optimum energy source scheduling relying on a model-free strategy using reinforcement learning to achieve minimum energy cost consumption. The proposed methodology is verified using improved PSO. Simulations and theoretical calculations reveal that the proposed scheme is very successful at decreasing energy consumption costs, in addition to meeting the preferences of users. [ABSTRACT FROM AUTHOR]

The energy management system (EMS) of hybridization and electrification plays a pivotal role in improving the stability and cost-effectiveness of future vehicles. Existing efforts mainly concentrate on specific optimization targets, like fuel consumption, without sufficiently taking into account the degradation of on-board power sources. In this context, a novel multi-objective energy management strategy based on deep reinforcement learning is proposed for a hybrid electric vehicle (HEV), explicitly conscious of lithium-ion battery (LIB) wear. To be specific, this paper mainly contributes to three points. Firstly, a non-parametric reward function is introduced, for the first time, into the twin-delayed deep deterministic policy gradient (TD3) strategy, to facilitate the optimality and adaptability of the proposed energy management strategy and to mitigate the effort of parameter tuning. Then, to cope with the problem of state redundancy, state space refinement techniques are included in the proposed strategy. Finally, battery health is incorporated into this multi-objective energy management strategy. The efficacy of this framework is validated, in terms of training efficiency, optimality and adaptability, under various standard driving tests. [ABSTRACT FROM AUTHOR]

As an effective reinforcement learning (RL) algorithm, Q-learning has been applied to energy management strategy of hybrid electric vehicle (HEV) in recent years. In the existing literatures, the values of three hyperparameters based on Q-learning are all given in advance, which are respectively exploratory rate ε, discount factor γ and learning rate α. However, different values of hyperparameters will influence on fuel economy of the vehicle and offline computation speed. In this paper, it is proposed that the method of optimization on hyperparameters adapting to driving cycle. Firstly, the mathematical model between three hyperparameters and iteration times is established based on inherent regularity of hyperparameters influencing on vehicle performance respectively. Secondly, it is determined that the optimal changing index k_index of iteration number based on Q-learning corresponding to typical driving cycles. Finally, the simulation model of Yubei District in Chongqing is constructed based on the method of Kullback-Leibler (KL) driving cycle identification. The simulation results indicate that equivalent fuel consumption of the proposed strategy is reduced by 0.4 % and the offline operation time is reduced by 6 s. It can be concluded that the proposed strategy can not only improve fuel economy of the vehicle, but also accelerate the computation speed. [ABSTRACT FROM AUTHOR]

Wireless charging vehicle to grid (V2G) system is not-so-futuristic. It can maintain the power supply balance and stabilize the grid through a wireless link between vehicles using batteries based on energy supply-demand statistics. Most of the existing privacy preservation methods are based on data tampering approaches (e.g., encryption, adding noise, aggregation), which are not practicable over V2G due to complexity in billing and inaccurate state estimation. In this paper, we develop a novel adaptive demand-side energy management framework by employing federated learning-based privacy preservation for the wireless charging V2G systems. Our framework learns the temporal evolution of energy consumption of dynamic charging electric vehicles in a distributed fashion and exploits the reinforcement learning model for cost-saving and reward maximization. The convergence and the privacy preservation properties of the framework are demonstrated with extensive evaluations. [ABSTRACT FROM AUTHOR]

• TIP-EMS enhances EV energy efficiency by integrated prediction and energy management. • Significant driving economy boost; 1.6%-4.2% better than state-of-the-art PEMS. • TIP-EMS shows robust performance with varying prediction weight adjustments. To enhance the energy efficiency of electrified vehicles (EVs), developing effective energy management strategies (EMS) for hybrid storage systems is essential. Predictive EMS (PEMS) that foresee future vehicle speeds have demonstrated substantial potential in boosting EMS performance. However, traditional PEMS models, employing a sequential approach of speed prediction followed by energy allocation, are hampered by cumulative errors. These errors from the initial speed predss this issue, this paper introduces a novel solution: the trainable integrated preiction negatively impact the efficiency of subsequent energy management. To addrediction and energy management strategy (TIP-EMS). Contrasting with conventional sequential PEMS, TIP-EMS features a dual-branch, integrated neural network, which is fully trainable. This network processes driving status inputs via attention layers, with one branch dedicated to energy management objectives using a reinforcement learning (RL) algorithm, and the other to vehicle speed prediction. Both branches are trained simultaneously, but post-training, only the RL branch is activated for energy management. Implemented with a soft actor-critic RL algorithm, TIP-EMS is applied to a fuel cell EV for optimized energy management. The validation involved training TIP-EMS using 27 driving profiles, which developed its prediction and energy management capabilities, followed by tests in untrained scenarios. The results show that TIP-EMS surpasses conventional sequential PEMS by up to 4.2% in scenarios where prediction accuracies are comparable, highlighting the efficacy of the trainable integrated mechanism. In addition, TIP-EMS exhibits superior energy conservation compared to non-predictive RL strategies. Lastly, TIP-EMS exhibits robustness to adjustments in the weight given to the prediction objective, further confirming its practical applicability. [ABSTRACT FROM AUTHOR]

As a fundamental task of hybrid electric vehicles (HEVs), energy management strategies are critical for improving the performance. This paper proposes a new queue-Dyna reinforcement learning (RL) algorithm based energy management strategy (EMS), which can substantially reduce the online learning time while guarantees the control performance compared with widely used Q-learning. To solve the existing problems of direct and indirect RL based EMSs, a blended RL algorithm, Dyna, is introduced first. By reusing the actual experience to construct a model online, Dyna integrates direct and indirect RL, and thus has the advantages of both. Furthermore, two novel strategies of backward focusing and prioritized sweeping are incorporated in the Dyna framework, developing the queue-Dyna algorithm. To the best of our knowledge, it's the first attempt of adopting queue-Dyna in the EMS of a HEV. A comparative simulation of direct RL, indirect RL, Dyna and queue-Dyna is implemented, and the results demonstrate the proposed algorithm achieves a great improvement in fast learning and maintains satisfied fuel consumption. At last, a hardware-in-the-loop experiment verified the real-time performance of the proposed EMS. [ABSTRACT FROM AUTHOR]

• An integrated adaptive dynamic programming for data-driven optimal control is presented. • An advantage function is introduced in ADP to improve the control performance. • An integrated neural network is designed for the value function and control policy simultaneously. • LQR and DP are used as benchmarks to evaluate the control performance. • Three different cases are studied to indicate the optimality, robustness and practical application. In this paper a novel integrated adaptive dynamic programming method with an advantage function is developed to solve model-free optimal control problems and improve the control performance. The advantage function is utilized to evaluate the cost resulting from the action (control variables) which does not follow the optimal control policy. The Q function in Q-learning can thus be built from a value function and the advantage function. The control policy is then improved through minimizing the Q function. To employ the proposed algorithm, an integrated multi-layer neural network (INN) is designed for the value function and the control variables. Only one single neural network requires adaption. This avoids the iterative learning of two separate networks in the heuristic dynamic programming-based methods. Simulation for linear and non-linear optimal control problems is studied. Comparing to the optimal solutions resulting from the linear quadratic regulator and dynamic programming (DP), the proposed INN design can lead to closer control performance than the ones with action dependent heuristic dynamic programming (ADHDP). Furthermore INN is applied to optimize the energy management strategy of hybrid electric vehicles for fuel economy. The fuel consumption based on INN is lower than the one from ADHDP and much closer to the optimal results by DP. The result indicates the near fuel-optimality and an effective practical application. [ABSTRACT FROM AUTHOR]

Reducing emissions and energy consumption of autonomous vehicles is critical in the modern era. This paper presents an intelligent energy management system based on Reinforcement Learning (RL) for conventional autonomous vehicles. Furthermore, in order to improve the efficiency, a new exploration strategy is proposed to replace the traditional decayed ε-greedy strategy in the Q-learning algorithm associated with RL. Unlike traditional Q-learning algorithms, the proposed self-adaptive Q-learning (SAQ-learning) can be applied in real-time. The learning capability of the controllers can help the vehicle deal with unknown situations in real-time. Numerical simulations show that compared to other controllers, Q-learning and SAQ-learning controllers can generate the desired engine torque based on the vehicle road power demand and control the air/fuel ratio by changing the throttle angle efficiently in real-time. Also, the proposed real-time SAQ-learning is shown to improve the operational time by 23% compared to standard Q-learning. Our simulations reveal the effectiveness of the proposed control system compared to other methods, namely dynamic programming and fuzzy logic methods. [ABSTRACT FROM AUTHOR]

Energy management strategy (EMS) has a huge impact on the energy efficiency of hybrid electric vehicles (HEVs). Recently, fast-growing number of studies have applied different deep reinforcement learning (DRL) based EMS for HEVs. However, a unified performance review benchmark is lacking for most popular DRL algorithms. In this study, 13 popular DRL algorithms are applied as HEV EMSs. The reward performance, computation cost, and learning convergence of different DRL algorithms are discussed. In addition, HEV environments are modified to fit both discrete and continuous action spaces. The results show that the stability of agent during the learning process of continuous action space is more stable than discrete action space. In the continuous action space, SAC has the highest reward, and PPO has the lowest time cost. In discrete action space, DQN has the lowest time cost, and FQF has the highest reward. The comparison among SAC, FQF, rule-based, and equivalent consumption minimization strategies (ECMS) shows that DRL EMSs run the engine more efficiently, thus saving fuel consumption. The fuel consumption of FQF is 10.26% and 5.34% less than Rule-based and ECMS, respectively. The contribution of this paper will speed up the application of DRL algorithms in the HEV EMS application. • A propulsion model is designed for a Hybrid Electric Vehicle in python and the entire code is shared in Github. • 13 popular DRL algorithms are introduced and both discrete and continuous action space algorithms are considered. • Key measures, including convergence reward/episode/time and final test result, are compared. • The best DRL algorithms of each action space type are compared with baseline EMS. [ABSTRACT FROM AUTHOR]

To further improve the fuel economy of series hybrid electric tracked vehicles, a reinforcement learning (RL)-based real-time energy management strategy is developed in this paper. In order to utilize the statistical characteristics of online driving schedule effectively, a recursive algorithm for the transition probability matrix (TPM) of power-request is derived. The reinforcement learning (RL) is applied to calculate and update the control policy at regular time, adapting to the varying driving conditions. A facing-forward powertrain model is built in detail, including the engine-generator model, battery model and vehicle dynamical model. The robustness and adaptability of real-time energy management strategy are validated through the comparison with the stationary control strategy based on initial transition probability matrix (TPM) generated from a long naturalistic driving cycle in the simulation. Results indicate that proposed method has better fuel economy than stationary one and is more effective in real-time control. [ABSTRACT FROM AUTHOR]

The concept of smart grid (SG) has been introduced to improve the operation of the power systems. In modern structures of power systems, different reasons prompt researchers to suggest integrated analysis of multi-carrier energy systems. Considering synergy effects of the couplings between different energy carriers and utilising intelligent technologies for monitoring and controlling of energy flow may change energy system management in the future. In this paper, we propose a new solution which is entitled ‘smart energy hub’ (SEH) that models a multi-carrier energy system in a SG. SEH solutions allow homeowners to manage their energy consumption to reduce their electricity and gas bill. We present this concept for a residential customer by an ‘energy management system’ which uses reinforcement learning algorithm and Monte Carlo estimation method for finding a near optimal solution. The simulation results show that by using this concept and then applying the algorithm for a residential customer, running costs are reduced up to 40% while keeping the household owner's desired comfort levels. [ABSTRACT FROM PUBLISHER]

• Multi-agent dueling double deep Q-network optimization framework was established. • Prioritized experience replay and feasible action screening mechanism was applied. • Value-decomposition network was used to solve multi-agent reliability problems. • Simulation results prove the economy and doability of the method. Under the background of high global building energy consumption, meeting the ever-growing energy consumption demand of building energy system (BES) through renewable energy is one of the effective ways to promote the clean transformation of global energy structure and achieve "carbon neutrality". However, with the introduction of renewable energy, BES control becomes more complicated. The mismatch between supply and demand sides limits the further growth of renewable energy consumption, which is caused by fluctuation of renewable energy and randomness of load. Therefore, it is challenging to develop an efficient framework to realize the cooperative control of various controlled objects in supply and demand sides. To address this challenge, a multi-agent deep reinforcement learning framework was proposed to optimize the energy management of the building. In this paper, a dueling double deep Q-network was used for optimization of single agent, and value-decomposition network was put forward to solve the cooperation optimization of multiple agents. Also, considering the controlled characteristics of BES, prioritized experience replay and feasible action screening mechanism were introduced to accelerate the convergence and maintain stability of the algorithm applied to BES. Simulation results show that, the multi-agent cooperation algorithm can realize the control of variously different devices at the same time and achieve multi-objective cooperation optimization of BES. Moreover, the proposed approach reduced the uncomfortable duration by 84%, the unconsumed amount of renewable energy by 43%, and the energy cost by 8% compared with the conventional rule-based control approach. [ABSTRACT FROM AUTHOR]

Buildings need advanced control for the efficient and climate-neutral use of their energy systems. Model predictive control (MPC) and reinforcement learning (RL) arise as two powerful control techniques that have been extensively investigated in the literature for their application to building energy management. These methods show complementary qualities in terms of constraint satisfaction, computational demand, adaptability, and intelligibility, but usually a choice is made between both approaches. This paper compares both control approaches and proposes a novel algorithm called reinforced predictive control (RL-MPC) that merges their relative merits. First, the complementarity between RL and MPC is emphasized on a conceptual level by commenting on the main aspects of each method. Second, the RL-MPC algorithm is described that effectively combines features from each approach, namely state estimation, dynamic optimization, and learning. Finally, MPC, RL, and RL-MPC are implemented and evaluated in BOPTEST, a standardized simulation framework for the assessment of advanced control algorithms in buildings. The results indicate that pure RL cannot provide constraint satisfaction when using a control formulation equivalent to MPC and the same controller model for learning. The new RL-MPC algorithm can meet constraints and provide similar performance to MPC while enabling continuous learning and the possibility to deal with uncertain environments. • MPC and RL are compared and merged for the same building optimal control problem. • RL-MPC is an algorithm that combines methods from machine learning and control theory. • MPC, RL, and RL-MPC are evaluated and benchmarked in the BOPTEST simulation framework. • MPC effectively uses the controller model while pure RL violates the constraints. • RL-MPC enables learning and meets the constraints with similar performance to MPC. [ABSTRACT FROM AUTHOR]

A novel deep reinforcement learning (DRL) control framework for the energy management strategy of the series hybrid electric tracked vehicle (SHETV) is proposed in this paper. Firstly, the powertrain model of the vehicle is established, and the formulation of the energy management problem is given. Then, an efficient deep reinforcement learning framework based on the double deep Q-learning (DDQL) algorithm is built for the optimal problem solving, which also contains a modified prioritized experience replay (MPER) and an adaptive optimization method of network weights called AMSGrad. The proposed framework is verified by the realistic driving cycle, then is compared to the dynamic programming (DP) method and the previous deep reinforcement learning method. Simulation results show that the newly constructed deep reinforcement learning framework achieves higher training efficiency and lower energy consumption than the previous deep reinforcement learning method does, and the fuel economy is proved to approach the global optimality. Besides, its adaptability and robustness are validated by different driving schedules. • The powertrain model of the series hybrid electric tracked vehicle is established. • A new control framework based on double deep Q-learning algorithm is constructed. • Modified prioritized experience replay is designed to improve training efficiency. • An adaptive optimization method is applied to update weights of the neural network. • The proposed deep reinforcement learning framework realizes better performance. [ABSTRACT FROM AUTHOR]

Grid-connected microgrids consisting of renewable energy sources, battery storage, and load require an appropriate energy management system that controls the battery operation. Traditionally, the operation of the battery is optimised using 24 h of forecasted data of load demand and renewable energy sources (RES) generation using offline optimisation techniques, where the battery actions (charge/discharge/idle) are determined before the start of the day. Reinforcement Learning (RL) has recently been suggested as an alternative to these traditional techniques due to its ability to learn optimal policy online using real data. Two approaches of RL have been suggested in the literature viz. offline and online. In offline RL, the agent learns the optimum policy using predicted generation and load data. Once convergence is achieved, battery commands are dispatched in real time. This method is similar to traditional methods because it relies on forecasted data. In online RL, on the other hand, the agent learns the optimum policy by interacting with the system in real time using real data. This paper investigates the effectiveness of both the approaches. White Gaussian noise with different standard deviations was added to real data to create synthetic predicted data to validate the method. In the first approach, the predicted data were used by an offline RL algorithm. In the second approach, the online RL algorithm interacted with real streaming data in real time, and the agent was trained using real data. When the energy costs of the two approaches were compared, it was found that the online RL provides better results than the offline approach if the difference between real and predicted data is greater than 1.6%. [ABSTRACT FROM AUTHOR]

Demand response (DR) has been acknowledged as an effective method to improve the stability, and financial efficiency of power grids. During operation of a DR program, there are usually multiple interactions among different grid entities, which complicates decision-making processes with respect to grid operations. Recently, reinforcement learning (RL) has attracted increasing attention for managing complex decision-making problems, owing to its self-learning capacity. Several theoretical RL-based approaches have been proposed for addressing various DR issues, but the practical feasibility of these theoretical approaches remains to be proven. In this paper, a conceptual architecture is firstly proposed to support DR management of a diversified facility in the context of a price-based DR environment. Secondly, exhaustive guidelines are provided to illustrate how to implement a multi-agent RL-based algorithm in the constructed DR management system. Afterwards, a laboratory-level testbed was set up to evaluate the effectiveness of the deployed DR algorithm. The experimental evaluation results show that the RL-based DR algorithm takes about 20s and 50 episodes to achieve optimal load control policy. By executing the optimal operation policy, the overall energy consumption during the highest price period (i.e., 15:00–18:00) is significantly reduced by 133.6% compared with the lowest price period (i.e., 02:00–05:00). • Explore reinforcement learning (RL-) based approaches by practical implementation. • Construct an RL-based demand response (DR) oriented experimental facility. • Provide exhaustive guidelines for practical implementation of an RL-based DR system. • The feasibility of a RL approach in practical DR implementation is fully assessed. [ABSTRACT FROM AUTHOR]

Many studies have proposed a peer-to-peer energy market where the prosumers' actions, including energy consumption, charge and discharge schedule of energy storage systems, and transactions in local energy markets, are controlled by a central operator. In this paper, prosumers' actions are not controlled by an operator, and the prosumers freely participate in the local energy market to trade energy with other prosumers. We designed and modeled a local energy market with a management algorithm that uses community energy storage for prosumers who competitively participate in trade in the real-time energy market. We propose an energy-trade management algorithm that manages the trades of prosumers in two phases based on bids and offers submitted by prosumers. The first phase is to manage the trade of prosumers who have submitted fair prices to trade with other prosumers in the real-time energy market. The second phase is managing the trade of prosumers that could not trade in the first phase. Community energy storage is employed in the second phase and controlled by a reinforcement learning-based trading algorithm to decide whether to buy, sell, or do nothing with the prosumers. The action of buying and selling means charging and discharging the community energy storage, respectively. Numerical results show that the proposed trading algorithm gains a near-maximum profit. Besides, we verified that community energy storage yields more profit than the battery wear-out cost. [ABSTRACT FROM AUTHOR]

This paper proposes a privacy-preserving energy management of a shared energy storage system (SESS) for multiple smart buildings using federated reinforcement learning (FRL). To preserve the privacy of energy scheduling of buildings connected to the SESS, we present a distributed deep reinforcement learning (DRL) framework using the FRL method, which consists of a global server (GS) and local building energy management systems (LBEMSs). In the framework, the LBEMS DRL agents share only a randomly selected part of their trained neural network for energy consumption models with the GS without consumer's energy consumption data. Using the shared models, the GS executes two processes: (i) construction and broadcast of a global model of energy consumption to the LBEMS agents for retraining their local models and (ii) training of the SESS DRL agent's energy charging and discharging from and to the utility and buildings. Simulation studies are conducted using one SESS and three smart buildings with solar photovoltaic systems. The results demonstrate that the proposed approach can schedule the charging and discharging of the SESS and an optimal energy consumption of heating, ventilation, and air conditioning systems in smart buildings under heterogeneous building environments while preserving the privacy of buildings' energy consumption. [ABSTRACT FROM AUTHOR]

• The intelligent SOC allocation strategy works well in variable scenario. • Segment-based planning reduces the drawback of uncertain traffic information. • Reinforcement learning ensures the optimization effect and adaptive-update ability. At present, plug-in hybrid electric vehicle has got widely concerned and studied for its special advantages of fuel-saving and emission-reduction potential. In order to improve the electric energy efficiency in power battery and meet the practical requirements of energy management strategies, an intelligent SOC-consumption allocation of commercial plug-in hybrid electric vehicles in variable scenario is proposed in this paper. Generated by random combination of the selected driving cycle units, the variable scene could reflect the driving situation of commercial vehicles performing transportation tasks at some extent. The strategy is trained and generated by a reinforcement learning framework. Firstly, a power-flow analysis model is constructed. Then, a dynamic programming description that could solve the optimal power-flow is defined. Furthermore, the optimizations for different initial conditions were performed repeatedly. With the solution results, an energy-conversion model is generated to describes the conversion of fuel and electricity, and reward in reinforcement learning was defined and used to guide the agent to improve and enhance the required SOC-allocation strategy. The trained strategy performed a reasonable SOC-allocation according to the cycle unit and residual SOC; and fuel consumption under average condition is significantly lower than that of two normally used SOC-allocation strategies. [ABSTRACT FROM AUTHOR]

Currently, many intelligent building energy management systems (BEMSs) are emerging for saving energy in new and existing buildings and realizing a sustainable society worldwide. However, installing an intelligent BEMS in existing buildings does not realize an innovative and advanced society because it only involves simple equipment replacement (i.e., replacement of old equipment or LED (Light Emitting Diode) lamps) and energy savings based on a stand-alone system. Therefore, artificial intelligence (AI) is applied to a BEMS to implement intelligent energy optimization based on the latest ICT (Information and Communications Technologies) technology. AI can analyze energy usage data, predict future energy requirements, and establish an appropriate energy saving policy. In this paper, we present a dynamic heating, ventilation, and air conditioning (HVAC) scheduling method that collects, analyzes, and infers energy usage data to intelligently save energy in buildings based on reinforcement learning (RL). In this regard, a hotel is used as the testbed in this study. The proposed method collects, analyzes, and infers IoT data from a building to provide an energy saving policy to realize a futuristic HVAC (heating system) system based on RL. Through this process, a purpose-oriented energy saving methodology to achieve energy saving goals is proposed. [ABSTRACT FROM AUTHOR]

A reinforcement-learning-based energy management strategy is proposed in this paper for managing energy system of Fuel Cell Hybrid Electric Vehicles (FCHEV) equipped with three power sources. A hierarchical power splitting structure is employed to shrink large state-action space based on an adaptive fuzzy filter. Then, the reinforcement-learning-based algorithm using Equivalent Consumption Minimization Strategy (ECMS) is proposed for tackling high-dimensional state-action space, and finding a trade-off between global learning and real-time implementation. The power splitting policy based on experimental data is obtained by using reinforcement learning algorithm, which allows for many different driving cycles and traffic conditions. The proposed energy management strategy can achieve low computation cost, optimal fuel cell efficiency and energy consumption economy. Simulation results confirm that, compared with existing learning algorithms and optimization methods, the proposed reinforcement-learning-based energy management strategy using ECMS can achieve high computation efficiency, lower power fluctuation of fuel cell and optimal fuel economy of FCHEV. • Decouple power demand of FCHEV based on fuzzy filter to improve RL algorithm. • Propose a novel algorithm to achieve optimal EMS for splitting power of sources. • Combine ECMS to refine algorithm to optimize fuel consumption and fuel cell lifespan. • Utilize experimental data to confirm effectiveness of the proposed algorithm. [ABSTRACT FROM AUTHOR]

The optimization and training processes of deep reinforcement learning (DRL) based energy management strategy (EMS) can be very slow and resource-intensive. In this paper, an improved energy management framework that embeds expert knowledge into deep deterministic policy gradient (DDPG) is proposed. Incorporated with the battery characteristics and the optimal brake specific fuel consumption (BSFC) curve of hybrid electric vehicles (HEVs), we are committed to solving the optimization problem of multi-objective energy management with a large space of control variables. By incorporating this prior knowledge, the proposed framework not only accelerates the learning process, but also gets a better fuel economy, thus making the energy management system relatively stable. The experimental results show that the proposed EMS outperforms the one without prior knowledge and the other state-of-art deep reinforcement learning approaches. In addition, the proposed approach can be easily generalized to other types of HEV EMSs. • A rule-interposing DRL-based energy management system is proposed. • The embedded knowledge is utilized to solve the optimization problems of EMS. • Extensive comparative experiments between RI DDPG and RI DQL are conducted. • The average fuel economy of RI DDPG is reduced by 8.9% than RI DQL, reaching 93.8% of DP's. [ABSTRACT FROM AUTHOR]

• The overall model for the hybrid electric tracked vehicle is built in detail. • Fast Q -learning algorithm is applied to derive energy management strategy. • An efficient online energy management strategy update framework is constructed. • Hardware-in-loop simulation experiment is conducted to validate the performance. • The strategy improves fuel economy and has potential for real-time applications. The energy management approach of hybrid electric vehicles has the potential to overcome the increasing energy crisis and environmental pollution by reducing the fuel consumption. This paper proposes an online updating energy management strategy to improve the fuel economy of hybrid electric tracked vehicles. As the basis of the research, the overall model for the hybrid electric tracked vehicle is built in detail and validated through the field experiment. To accelerate the convergence rate of the control policy calculation, a novel reinforcement learning algorithm called fast Q -learning is applied which improves the computational speed by 16%. The cloud-computation is presented to afford the main computation burden to realize the online updating energy management strategy in hardware-in-loop simulation bench. The Kullback-Leibler divergence rate to trigger the update of the control strategy is designed and realized in hardware-in-loop simulation bench. The simulation results show that the fuel consumption of the fast Q -learning based online updating strategy is 4.6% lower than that of stationary strategy, and is close to that of dynamic programming strategy. Besides, the computation time of the proposed method is only 1.35 s which is much shorter than that of dynamic programming based method. The results indicate that the proposed energy management strategy can greatly improve the fuel economy and have the potential to be applied in the real-time application. Moreover, the adaptability of the online energy management strategy is validated in three realistic driving schedules. [ABSTRACT FROM AUTHOR]