Search

Human-in-the-loop for reinforcement learning (RL) is usually employed to overcome the challenge of sample inefficiency, in which the human expert provides advice for the agent when necessary. The current human-in-the-loop RL (HRL) results mainly focus on discrete action space. In this article, we propose a $Q$ value-dependent policy (QDP)-based HRL (QDP-HRL) algorithm for continuous action space. Considering the cognitive costs of human monitoring, the human expert only selectively gives advice in the early stage of agent learning, where the agent implements human-advised action instead. The QDP framework is adapted to the twin delayed deep deterministic policy gradient algorithm (TD3) in this article for the convenience of comparison with the state-of-the-art TD3. Specifically, the human expert in the QDP-HRL considers giving advice in the case that the difference between the twin $Q$ -networks’ output exceeds the maximum difference in the current queue. Moreover, to guide the update of the critic network, the advantage loss function is developed using expert experience and agent policy, which provides the learning direction for the QDP-HRL algorithm to some extent. To verify the effectiveness of QDP-HRL, the experiments are conducted on several continuous action space tasks in the OpenAI gym environment, and the results demonstrate that QDP-HRL greatly improves learning speed and performance.

State-of-charge (SoC) balancing in distributed energy storage systems (DESS) is crucial but challenging. Traditional deep reinforcement learning approaches struggle with real-world multiagent cooperation for SoC balance in these decentralized systems. To address these significant hurdles, this article pioneers an innovative fully-decentralized multiagent reinforcement learning (FDMARL) strategy, specifically tailored for DESS. First, the SoC balancing problem is formulated into a finite decentralized Markov decision process with action constraints derived from the microgrid context. Then, the average consensus algorithm is introduced for expanding the agent's observation and obtaining global information through a communication network. To improve multiagent system cooperation, a novel demand balance algorithm is proposed to refine agent actions for precise demand distribution. By the above modules, the FDMARL reveals outstanding performance in a fully-decentralized system without any expert experience or modeling. Finally, numerous simulations were carried out on pymgrid, (i.e., a Python-based open-source microgrid platform), which shows that: The FDMARL exceeds traditional reinforcement learning in decentralized cooperation, effectively manages large-scale systems with random states and demand/PV series, and maintains robustness with ESU failure or integration.

Physics-informed machine learning (PIML) has proven to be a valuable approach for overcoming data scarcity challenges by incorporating physical models into machine learning methods. However, PIML faces limitations in handling complex spatial relationships, as its process information is obtained from disordered collocation points. Fuzzy systems, based on expert knowledge, can provide an interpretable way for tackling strong process nonlinearities. This article proposes a brand-new physics-informed spatial fuzzy system framework (PiFuz) to capture the essential system information of complex distributed parameter systems. PiFuz utilizes spatial membership functions to transform collocation points into a 3-D fuzzy input. This input is processed by the inference mechanism, leveraging its 3-D nature to produce fuzzy outputs with distinctive spatial characteristics. A feature fusion module is utilized to integrate these characteristics and generate the distributed system state. Utilizing the known physical knowledge base, the proposed framework undergoes automatic tuning while preserving process interpretability, resulting in an optimal model that aligns with the actual physical process. A reliable prediction of strong spatial nonlinear behaviors is achieved without the dependency of process data. For modeling higher dimensional spatiotemporal problems, the extension, a multikernel PiFuz framework (MKPiFuz), is further developed to improve the representation of heterogeneous time-varying nonlinear behaviors. By incorporating spatial and wavelet kernels, MKPiFuz extracts underlying features from spatial and temporal dimensions, respectively. Experimental investigations on thermal process of the battery module demonstrate the good accuracy in modeling complex spatiotemporal systems.

The charging process design is crucial for optimizing the performance of lithium-ion batteries by identifying protocols that meet diverse demands. The main challenges include: 1) the high costs of battery experiments; 2) the multiple user preferences associated with the demands; and 3) the intricate high-dimensional search space of charging protocols. In light of this, this article presents a Gaussian process-accelerated multiobjective evolutionary design method for effective charging process design. To resolve the first concern, an electrochemical-thermal-aging model is constructed to evaluate charging protocols precisely, substituting the need for expensive battery experiments. Besides, the Gaussian process is applied to accelerate the evaluation process further. Regarding the second issue, a Gaussian process-accelerated two-archive evolutionary algorithm (GPA-TAEA) is developed to efficiently search for a set of optimal charging protocols that satisfy multiple user preferences. To address the third challenge, differential evaluation—an evolutionary algorithm proven effective for large-scale optimization—is employed to enhance the search process. The simulation results demonstrate that: 1) the proposed method effectively reduces the time required for charging process design, yielding a collection of optimal charging protocols that includes 530 solutions within 1000 simulations; 2) compared with five other multiobjective optimization algorithms, GPA-TAEA exhibits superior convergence and diversity; and 3) compared with fixed preference-based methods, GPA-TAEA demonstrates greater efficiency, saving 54% in simulation evaluations and 70% in time when considering seven user preferences.

Modeling the temperature distribution of a battery is critical to its safe operation. Data-based modeling methods are computationally efficient, but require a large number of sensors, whereas physics-based modeling methods have better generalization, but the unknown dynamics of the actual scene are ignored. A physics-dominated neural network is presented to integrate electric–thermal mechanism of the battery and data information through a weight adaptive function. The electric–thermal coupling equation of the battery under complex conditions is taken as the prior knowledge to update parameters of the network, whereas the characteristic data obtained by the unique sensor are used to compensate the unknown disturbance in the actual scene. A well-trained model can predict the temperature distribution of the battery over entire space with a single sensor, and can also provide reasonable predictions for longer periods of time under extreme conditions. Experiments show that the proposed method outperforms traditional methods that rely only on pure data or pure physics.

Unknown distributed incipient thermal fault detection and localization are vital to the safe operation of batteries while they have not been given sufficient attention in existing works compared to the studies on estimation of State of Charge (SoC) as well as State of Health (SoH). In order to fill this gap, a backstepping-based fault localization filter (FLF) is presented. Generally, full-state temperature measurement is required to achieve fault localization, which is impossible in applications. However, with the help of interpolation-based approximation, the required number of sensors decreases from infinity to only a few, which guarantees the usability of FLF. A comprehensive methodology framework, including the FLF design, residual evaluation in a distributed manner, and threshold computation, is introduced to guarantee reliable and robust performance in a real-time pattern. Theoretic analysis as well as experiment validations are presented to for validation.