Search

Reinforcement learning-based methods have addressed production scheduling problems with flexible processing constraints. However, delayed rewards arise due to the dynamic arrival of jobs and transportation constraints between two successive operations. The flow time of operations can only be determined after processing due to the possibility that the solution for job sequencing may change if new operations are inserted in dynamic environments. Job sequencing is often overlooked in single-agent-based scheduling methods. The lack of information sharing between multiple agents necessitates that researchers manually design reward functions to fit the relationship between optimization objectives and rewards, thereby reducing the accuracy of the learned policies. Thus, this paper proposes a multi-agent-based scheduling optimization framework that facilitates collaboration between the agents of both machines and jobs to address dynamic flexible job-shop scheduling problems (DFJSP) with transportation time constraints. Then, this paper formulates the Partial Observation Markov Decision Process and constructs a reward-sharing mechanism to tackle the delayed reward issue and facilitate policy learning. Finally, we develop an improved multi-agent dueling double deep Q network algorithm to optimize scheduling policy during long-term training. The results show that, compared with the state-of-the-art methods, the proposed method efficiently shortens the weighted flow time under the trained and unseen scenarios. Additionally, the case study results demonstrate its efficiency and responsiveness. It indicates that the proposed method efficiently addresses production scheduling problems with complex constraints, including the insertion of jobs, transportation time constraints, and flexible processing routes. [ABSTRACT FROM AUTHOR]

With the deepening of modernization and industrialization, the issues of water pollution and scarcity have become more pressing. To address these issues, many wastewater treatment factories have been built to improve the reuse of water resources. However, the control of the wastewater treatment process (WWTP) is a complex task due to the highly nonlinear and strongly coupled nature. It is challenging to develop the accurate mechanism models of the wastewater treatment system. The improvement of the efficiency for the WWTP is crucial to safeguard the urban ecological environment. In this paper, adaptive critic with weight allocation (ACWA) is developed to address the optimal control problem in the WWTP. Different from the previous methods of the WWTP, system modeling is not adopted in this paper, which meets the actual physical background of the wastewater treatment system to a great extent. In addition, the actor-critic algorithm in reinforcement learning is used as the basic structure in the ACWA. It is worth noting that a novel weighted action-value function and the advantage function are introduced in the weight updating process of the action network and the critic network. The experimental results show that the control accuracy of the ACWA is greatly improved compared with the previous control methods. [ABSTRACT FROM AUTHOR]

In the domain of complex urban traffic networks, real-time Traffic Signal Control (TSC) serves as a pivotal strategy for mitigating congestion. Coordinating signal control across multiple intersections involves considerable complexity. Deep Reinforcement Learning (DRL) has emerged as a robust solution. In recent years, there has been rapid advancement in TSC methods, with numerous researchers employing various novel DRL methodologies. Yet, existing surveys lack timeliness and universality in capturing the latest research. There is a notable gap in current research surveys with respect to the latest developments in TSC. Therefore, the focus of this paper lies in analyzing the most recent papers from the past five years, with the aim to provide a comprehensive and multi-dimensional review of the evolution of DRL in TSC. The survey categorizes current research based on model setups, utilized algorithms, and application scenarios. Finally, this paper highlights potential directions for future TSC research. [ABSTRACT FROM AUTHOR]

Robotic force-based compliance control is a preferred approach to achieve high-precision assembly tasks. When the geometric features of assembly objects are asymmetric or irregular, reinforcement learning (RL) agents are gradually incorporated into the compliance controller to adapt to complex force-pose mapping which is hard to model analytically. Since force-pose mapping is strongly dependent on geometric features, a compliance controller is only optimal for current geometric features. To reduce the learning cost of assembly objects with different geometric features, this paper is devoted to answering how to reconfigure existing controllers for new assembly objects with different geometric features. In this paper, model-based parameters are first reconfigured based on the proposed Equivalent Theory of Compliance Law (ETCL). Then the RL agent is transferred based on the proposed Weighted Dimensional Policy Distillation (WDPD) method. The experiment results demonstrate that the control reconfiguration method costs less time and achieves better control performance, which confirms the validity of proposed methods. [ABSTRACT FROM AUTHOR]

The design of high-performance analog circuits is a time-consuming process that requires the involvement of highly qualified designers and frequent tedious interactions with circuit simulators. This sets a need for a method that accelerates the circuit design process and reduces unnecessary designer-simulator interactions. Existing methods are either slow or unable to deal with complex tasks. This paper proposes a simulation-based optimization method based on a deep reinforcement learning (DRL) agent to optimize analog circuits and accelerate the design process. In this method, the DRL agent exploits the soft-actor-critic algorithm based on an improved reward function considering multiple conditions to expedite the parameters optimization step of analog circuits. The designer provides the RL agent with the circuit testbench, design requirements and circuit parameters. The agent then repetitively interacts with the circuit simulator to tune these parameters. In this work, an incremental RL approach is also proposed to deal with complex circuits. Incrementality is found to be increasingly beneficial as the circuit complexity increases (increased number of design parameters and competing objectives). The effectiveness of the proposed method was validated on a complex circuit defined by a design space comprising as many as 36 dimensions (17 design variables and 19 competing objectives). The reported results demonstrate that our method can significantly accelerate the design process and yield high-performance optimized designs. Notably, the required number of simulation steps is on the order of 1e3, which is one to two orders of magnitude more efficient than similar state-of-the-art DRL approaches. • The paper proposes a novel approach to accelerate circuit design. • It is based on incremental reinforcement learning with an improved reward function. • It optimizes the design process and reduces designer-simulator interactions. • The approach is validated through two case studies in analog circuit design. • It achieves high-performance designs in less steps compared to similar methods. [ABSTRACT FROM AUTHOR]

Maintenance planning of networked multi-asset systems is a complex problem due to the inherent individual and collective asset constraints and dynamics as well as the size of the system and interdependencies among assets. Although multi-asset systems have been studied numerous times in the past decades, maintenance planning implications of the system's network characteristics have been barely analysed. Likewise, solutions that consider the network perspective suffer from scalability issues as a network-wide observability is assumed. This paper proposes a network maintenance planning approach based on the decomposition of the multi-asset network into fixed-size localised subnetworks. The overall network maintenance plan is produced by aggregating the subnetwork maintenance plans, which are computed independently via a multi-agent deep reinforcement learning (MARL) algorithm. The results are evaluated against a network-wide approach as well as the commonly-used individual approach. The paper also introduces a systematic approach to integrate the MARL resulting policy in a multi-asset agent-based model. Simulation results of several random asset networks and a large nationwide network infrastructure show that, although a network-wide approach outperforms, on average, other approaches considered, the localised subnetworks approach, provides an acceptable alternative in networks with small-world properties, without the need of a network-wide view. • A Network-wide maintenance approach outperforms individual approaches in networked assets with the topologies analysed. • In analysed networks with small-world properties, the Localised approach shows performance close to the network-wide approach. • A systematic integration of MARL and multi-asset agent-based modelling enables evaluation of maintenance approaches. [ABSTRACT FROM AUTHOR]

Since the multi-time scales and strong nonlinearity are objectively existed in the practical industrial processes, it is extremely challenging to design control policy for optimizing the industrial operation. Therefore, this paper focuses on the multi-time scale optimal operational control (OOC) problem with unmodeled dynamics. Under the framework of reinforcement learning (RL) technology, a self-learning model-free method by integrating zero-sum game and singular perturbation (SP) theories is proposed in this paper, such that composite controllers with signal compensation can be found under which the suboptimal operation of the whole process can be achieved using only measured data. The remarkable highlight in this paper is the novel design method of the composite controllers with signal compensations for unknown nonlinear systems with multi-rate operations. Finally, the effectiveness of the proposed method is verified by a practical example and a numerical example. [ABSTRACT FROM AUTHOR]

Nowadays, economic and environmental concerns in production have become increasingly significant. To address these issues, the Combined Economic and Emission Dispatch (CEED) problem has been introduced to optimize the power generation process by considering fuel cost and emitted substances. However, due to the nonlinearity and nonconvexity of the objective function, the optimization of CEED remains a challenge. In this paper, we develop a Reinforcement Learning-based Adaptive Differential Evolution (RLADE) algorithm to enhance the optimization performance. The mutation strategy and crossover probability of RLADE are optimized using Reinforcement Learning (RL) to respectively ensure better convergence speed and searchability. Additionally, two modifications of RL, namely the adaptive population size-based state division and fitness-ranking-based reward mechanism, are proposed to improve the accuracy of state division and reward calculation in RL. The experiments conducted in this paper consider two objective formulation methods of CEED problems, namely the quadratic and cubic criterion functions. The mean values and standard deviations of the obtained solutions were utilized to assess the performance of RLADE, as well as other comparative algorithms, namely DE algorithm and two RL-based DE variants. The results clearly demonstrate that RLADE surpasses its counterparts with proportion of 100%, 85.7%, and 100% for the 6-unit and 11-unit quadratic CEED problems, as well as cubic criterion functions, in terms of both search accuracy and convergence ability. Furthermore, the significance of RLADE's superiority is confirmed through the Wilcoxon's signed rank test. [ABSTRACT FROM AUTHOR]

This paper solves a bi-level optimization problem in which at the first level, the distribution system operator (DSO) as a retailer tends to extract selling energy price signals so that increases its profit and reduces the peak-to-average ratio (PAR) of the MMG transactive energy; and at the second level energy management problems of non-cooperative networked microgrids (MGs) are solved individually to minimize their cost. This game theoretically problem is known as the Stackelberg game and has been solved with classical optimization methods like Karush–Kuhn–Tucker (KKT) method previously. To solve this problem with the classical KKT method, the MGs' information should be provided for the DSO as a game starter, which is not the MGs' desire for their privacy concerns. With the aim of preserving the privacy of MGs, this paper has developed a decision-making mechanism based on reinforcement learning (RL) to help the DSO extract energy prices individually for each MG. Also, the deep neural network (DNN) is used as a practical tool to predict MGs' behavior in response to signal price. Furthermore, in the model used in this study, energy storage systems (ESSs) are dedicated to MGs which makes the model non-linear, thus the proposed method unlike the classic method is able to solve such a problem. The results obtained from the comparison between the classic and proposed method show that the implementation of this machine learning-based method is not only accurate enough but also is faster than the classical method, in addition to its privilege of privacy-preserving. Finally, a sensitivity analysis has been conducted to evaluate the DSO profit and PAR under different weighting factors of the objective function to obtain an appropriate performance. • Presenting a non-cooperative framework using deep reinforcement learning. • Considering cost minimization, profit maximization and PAR reduction simultaneously. • The proposed transactive energy model ensures the privacy of MGs. • Providing sensitivity analysis on the Monte Carlo algorithm parameters. [ABSTRACT FROM AUTHOR]

Possessing efficient supervisory control systems is crucial for maintaining the desired operational performance of complex industrial processes. Several challenges face the developers of these systems, such as requiring accurate physical models, dealing with the variability and uncertainty of process operating conditions and coordinating between local controllers to reach desired global performance. This paper proposes an intelligent supervisory control approach based on causal reinforcement learning (CRL) to effectively manipulate the controllers' setpoints of the process in a way that optimizes its key performance indicators (KPIs), thereby improving the energy efficiency of the process. The approach adopts deep reinforcement learning (DRL) to develop an efficient control policy through interaction with a process simulation. The DRL training history is then exploited using interpretable machine learning and process mining to build a discrete event system (DES) model, in the form of a state-event graph. The DES model identifies causal relationships between events and provides interpretability to the control policy developed by the DRL method. The DES discovered is exploited as a Markov decision process to apply the Q-learning algorithm as a CRL supervisor. The supervisor incorporates causal knowledge into its training process, thus improving the DRL control policy developed and identifying the event paths that optimize the process's KPIs. The proposed approach is validated using two heat recovery systems in a pulp & paper mill. It successfully achieves a control policy that reduces energy consumption by up to 15.6% for the first system and 5.02% for the second, compared to the expert's baseline methods. • This paper proposes a novel supervisory control approach for industrial processes. • The approach is based on causal reinforcement learning (CRL). • The approach integrates simulation-based optimization and process mining. • It learns an optimal control policy for recommending controllers' setpoints. • The approach is validated on two heat recovery systems in a pulp & paper mill. [ABSTRACT FROM AUTHOR]

Unmanned Aerial Vehicle (UAV) maneuver strategy learning remains a challenge when using Reinforcement Learning (RL) in this sparse reward task. In this paper, we propose Subtask-Masked curriculum learning for RL (Sub Mas -RL), an efficient RL paradigm that implements curriculum learning and knowledge transfer for UAV maneuver scenarios involving multiple missiles. First, this study introduces a novel concept known as subtask mask to create source tasks from a target task by masking partial subtasks. Then, a subtask-masked curriculum generation method is proposed to generate a sequenced curriculum by alternately conducting task generation and task sequencing. To establish efficient knowledge transfer and avoid negative transfer, this paper employs two transfer techniques, policy distillation and policy reuse, along with an explicit transfer condition that masks irrelevant knowledge. Experimental results demonstrate that our method achieves a 94.8% success rate in the UAV maneuver scenario, where the direct use of reinforcement learning always fails. The proposed RL framework Sub Mas -RL is expected to learn an effective policy in complex tasks with sparse rewards. [ABSTRACT FROM AUTHOR]

The microgrid market is growing significantly due to several drivers, such as the need to lower greenhouse gas emissions by integrating higher shares of distributed renewable energy sources, falling costs of microgrid components, the need for more reliable power supply infrastructures, and new off-grid solutions to foster electricity access in developing economies. Coordinated management of the microgrid components is crucial for their effectiveness, and this can be very challenging when hosting solar or wind generation. This paper studies the energy management problem of a microgrid based on reinforcement learning algorithms. The advantage of using these algorithms against other optimization and machine learning techniques is that they do not need past experiences to learn a strategy. The learning is based on trial and error experiences, which facilitates its easy implementation to other microgrids while demonstrating their facility to be applied in real cases. In particular, this paper proposes an implementation for an Energy Management System (EMS) in microgrids using the Twin-Delayed Deep Deterministic Policy Gradient (TD3) algorithm. Moreover, it compares the proposed algorithm with the Deep Q-Network (DQN). This comparison evaluates the improvement over exploiting the continuous nature of the decision variables against a discretization of the same since the DQN cannot make actions over a continuous space. [ABSTRACT FROM AUTHOR]

In complex urban traffic scenarios, autonomous vehicles face significant challenges in adapting to diverse and dynamic traffic conditions. Reward-based reinforcement learning has emerged as an effective approach to tackle these challenges. This paper presents a novel method that combines deep reinforcement learning with automotive dynamics systems. Building upon the Double Deep Q-learning algorithm, our approach integrates a Recurrent Neural Network with Gated Recurrent Units to enhance the environmental exploration capabilities of autonomous vehicles. To obtain more precise reward values, we introduce a trajectory tracking algorithm based on a combination of proportional-integral-derivative control and feedforward control within the automotive dynamics system. The proportional-integral-derivative controller is utilized for longitudinal control, while the Error-Optimized feedforward controller enhances lateral control, thereby improving trajectory tracking accuracy. Finally, extensive simulation experiments are conducted to evaluate the proposed method, comparing it against other baseline methods in terms of vehicle following and lane-changing scenarios. The results demonstrate that our approach significantly improves both the reward values and control performance of the algorithm. [Display omitted] • Novel DDQN_PID_LEO algorithm: DDQN, PID, and lateral control fusion. • Realistic traffic simulations in UE4 for dynamic scenarios. • Enhanced vehicle control with minimized lateral errors. • DDQN reinforcement learning for rewarding collision avoidance. [ABSTRACT FROM AUTHOR]

Due to the increasing high penetration of Photovoltaic (PV), it brings great challenge for voltage control issue of distribution network. To address this problem, this paper presents an improved collaborative Multi-Agent Reinforcement Learning (MARL) approach for proactive voltage control, aimed at mitigating voltage violations and minimizing power losses. The self-attention mechanism is embedded into the multi-agent soft actor-critic (MASAC) algorithm to enhance the collaboration of multi-agent system, which can well improve the learning efficiency to ensure the voltage safety of Distribution Network. In addition, the proposed learning approach is implemented on IEEE 33-bus, 141-bus and 322-bus systems, and the simulation results reveal that the proposed approach can control the voltage into safety domain as well as reduce power losses. [ABSTRACT FROM AUTHOR]

Stable fault self-healing ensures the smooth operation of intelligent industrial manufacturing processes. However, current research, while capable of inferring fault propagation paths or individually controlling manufacturing processes, still faces some challenges and limitations in addressing issues within manufacturing processes. In particular, it is easy to overlook the impact of fault variables on normal causal relationships in the manufacturing process and how to achieve adaptive fault repair from the source of faults. The biological immune system possesses sensitive nodes and a neuro-endocrine-immune network with interconnected functionalities, bearing remarkable resemblance to the sensor arrays, information networks, and intelligent controller systems in modern self-healing control systems. These problem domains have significant practical implications in industrial manufacturing processes and urgently require new methods and technologies to address them. To address the issue of root cause fault self-healing in intelligent industrial manufacturing processes, this paper proposes an innovative method that utilizes root cause inference and bio-inspired heuristic reinforcement learning models to achieve fault self-healing in industrial processes. First, innovatively introduce the Variational Autoencoder (VAE) into the Time Convolutional Network (TCN) to construct the TCN-VAE network. By performing feature reconstruction, the network's feature extraction capability is enhanced, further exploring the relationships between latent variables, and consequently building a causal graph of faults. A multi-head attention mechanism is introduced into the network, and the inference process is quantitatively evaluated, thereby improving the generalization and accuracy of root cause inference. Root cause determination rules are established to identify the fundamental reasons for fault occurrence, ensuring self-healing from the source of faults. Next, we developed for the first time a fault self-healing model based on reinforcement learning, utilizing the immune repair process of effector T cells to characterize the fault self-healing process. It effectively adapts to eliminate various levels of faults based on the state transition process of antigens in T helper cells. Additionally, stability control analysis is performed on the proposed model. The average reward function curve and Sobol sensitivity index demonstrates the model's strong robustness in practical applications. The validation results from case studies on the Tennessee Eastman (TE) and Continuous Stirred Tank Reactor (CSTR) show that the average improvement in fault repair F1 score was 0.0941 and 0.105 respectively, and the fastest improvement in fault repair response time was 0.266s and 0.258s respectively. It was confirmed that the root cause inference and fault repair results align with the actual mechanisms of the manufacturing processes. The proposed method has been verified to achieve stable and accurate root cause fault repair in real-world intelligent industrial manufacturing systems, making a significant contribution to achieving stable and accurate root cause fault repair in intelligent industrial manufacturing systems. • TCN-VAE enables feature reconstruction and latent variable exploration for root cause reasoning. • Multi-head attention enhances generality and accuracy in root cause reasoning. • Effector T-cell immune-inspired RL adapts to eliminate faults effectively. • The fault self-healing model exhibits strong reliability and robustness. • Method validated on Tennessee Eastman and continuous stirred-tank reactor datasets. [ABSTRACT FROM AUTHOR]

Traditional robotic peg-in-hole assembly methods rely on complex contact state analysis. Reinforcement learning (RL) is gradually becoming a preferred method of controlling robotic peg-in-hole assembly tasks. However, the training process of RL is quite time-consuming because RL methods are always globally connected, which means all state components are assumed to be the input of policies for all action components, thus increasing action space and state space to be explored. In this paper, we first define continuous space serialized Shapley value (CS3) and construct a connection graph to clarify the correlativity of action components on state components. Then we propose a local connection reinforcement learning (LCRL) method based on the connection graph, which eliminates the influence of irrelevant state components on the selection of action components. The simulation and experiment results demonstrate that the LCRL method can achieve the same average reward as the traditional RL method in only 49% of episodes. In the final episode, the LCRL method's final reward is 35% higher than that of the traditional RL method, which guarantees the rapidity and stability of the assembly process. [ABSTRACT FROM AUTHOR]

The lighting requirements are subjective and one light setting cannot work for all. However, there is little work on developing smart lighting algorithms that can adapt to user preferences. To address this gap, this paper uses fuzzy logic and reinforcement learning to develop an adaptive lighting algorithm. In particular, we develop a baseline fuzzy inference system (FIS) using the domain knowledge, generating light recommendation based on a set of intuitive rules. These rules, derived from existing literature, are based on environmental conditions i.e. daily glare index, and user information including age, activity, and chronotype. Through a feedback mechanism, the user interacts with the algorithm, correcting the algorithm output to their preferences. We interpret these corrections as rewards to a Q-learning algorithm, which tunes the FIS parameters online to match the user preferences. The Q-learning is a model-free learning algorithm that learns to act optimally by interacting with the user and the rewards it receives. This allows the proposed algorithm to work in a model-free manner, effectively handling the uncertainties that might arise from the individualistic preferences of users. To the authors' best knowledge, this algorithm is pioneering work in designing intelligent algorithms for personalized lighting control, featuring several elements of novelty, including the number of environmental and user-related inputs, the continuous control of light intensity as opposed to common on/off control, and the ability to learn user preferences. The algorithm is implemented in a real aircraft cabin and is evaluated in an extensive user study. The implementation results demonstrate that the developed algorithm possesses the capability to learn user preferences while successfully adapting to a wide range of environmental conditions and user characteristics. This underscores its viability as a potent solution for intelligent light management, featuring advanced learning capabilities. • Intelligent lighting algorithm with the ability to learn from and adapt to user preferences. • A fuzzy inference system tunes lighting based on environment and user traits. • The developed algorithm is tested and verified via an in-depth user study. [ABSTRACT FROM AUTHOR]

Grid-interactive building control poses a critical challenge in the context of grid modernization and decarbonization. Recently, various artificial intelligence-based optimal control approaches have been proposed, providing innovative solutions to optimal building control problems. However, researchers and practitioners are now confronted with a dilemma when selecting appropriate control strategies, ranging from model-based (knowledge-incorporating) to learning-based (data-driven) approaches, and hybrid methods combining them. Although each algorithm in existing literature claims superiority over specific baselines, their performance has never been systematically compared and analyzed, owing to the absence of a unified platform for comprehensive evaluation. To fill this knowledge gap, we identify and implement all state-of-the-art approaches within a modular training and evaluation framework, assessing their efficacy in a grid-interactive building control problem. In this paper, we also introduce a streamlined hybrid method that complements existing hybrid approaches. Our comparative study reveals and quantifies the advantages of hybrid methods: on average, they achieve near-optimal control while requiring less than 14% of the online computation of traditional model-based methods. To achieve this performance, they need as few as 2% of the training samples of purely learning-based methods. Finally, we provide insights into the merits, limitations, and implementation of each method to help researchers better understand the state of the art and future directions. The software framework implemented in this study is open-sourced to facilitate future research. [ABSTRACT FROM AUTHOR]

Similarity-based transfer learning for reinforcement learning has garnered attention for its potential to enhance target task learning. However, it faces significant challenges in efficiency and effectiveness, primarily stemming from issues such as sparse reward, long trajectory, and strict similarity. To solve these problems, this paper proposes a local instance-based transfer learning method for reinforcement learning. Instead of relying on sparse reward and long trajectory, this approach leverages the Q value of the local trajectory to evaluate similarity, thereby significantly enhancing transfer efficiency. Furthermore, by relaxing the strictness of the similarity, three transfer policies are proposed to facilitate positive transfer. Extensive experimental results demonstrate that the effectiveness and efficiency of the proposed method in comparison with traditional similarity-based transfer learning methods. [ABSTRACT FROM AUTHOR]

This paper delves into enhancing underwater images to improve human observation of underwater scenes. To this end, a reinforcement learning-based human visual perception-driven image enhancement paradigm for underwater scenes is proposed, referred to as INSPIRATION. This paradigm is so named because it constructs rewards for guiding model training by incorporating metrics inspired by human visual perception, eliminating the need for reference images. It overcomes the limitations of deep models, which often result in poor performance in dynamic and diverse underwater scenes due to the scarcity of underwater images with enhancement references. Specifically, a residual-enhancement network, composed of residual modules and a channel-attention module, is utilized to extract features as a state. An extensible collection encompassing diverse image enhancement algorithms is utilized to provide an image enhancement algorithm as an action. A multi-non-reference human visual perception metric increment is utilized as a reward, and proximal policy optimization (PPO) is utilized to conduct reinforcement learning. This paradigm treats underwater images as sole inputs during both training and implementation, learning and organizing a sequence of image enhancement algorithms to explicitly achieve a step-wise image enhancement process, aligning the enhanced images with human visual perception. Extensive qualitative and quantitative evaluations conclusively demonstrate that our paradigm outperforms nine state-of-the-art underwater image enhancement methods in terms of visual quality and achieves better performance on five underwater image quality assessment measures across three underwater image datasets. Additionally, the potential advantages of our paradigm as a pre-processing step in other underwater computer vision applications are demonstrated. The code has been released publicly for reproducibility and evaluations at https://gitee.com/wanghaoupc/uie_inspiration. [ABSTRACT FROM AUTHOR]

The scheduling of ship traffic in the port is an important operation that aims to improve the navigation efficiency of the port, which is greatly affected by a variety of unpredictable events. In line with this research direction, this paper explores the ship traffic scheduling problem (STSP) with uncertain arrival and departure times (STSP-UN). STSP-UN treats arrival and departure times as random variables, with the objective of generating a schedule plan that minimizes the total waiting time of ships in the port while accommodating all possible variations in arrival and departure times within predefined uncertainty sets. A robust counterpart of the deterministic STSP formulation is derived where the uncertain arrival and departure times are confined within budget uncertainty sets. To obtain solutions for the proposed model, a hybrid optimization method (MAVNS) is used, combining a memetic algorithm (MA) for global exploration with a variable neighborhood search algorithm (VNS) that utilizes problem-specific neighborhood operators for local search. Numerical experiments are carried out at the Comprehensive port in Huanghua based on various instance sizes. The results validate the rationality and effectiveness of both the proposed robust optimization model and the algorithm. Specifically, the solution produced by MAVNS is 44.40% and 27.65% superior to that of the genetic algorithm (GA) and MA, respectively, for small-scale instances. For large-scale instances, the improvement rate is 41.25% and 30.02% compared to GA and MA, respectively. The MAVNS algorithm is a superior approach compared to an adaptive heuristic algorithm based on reinforcement learning (GSAA-RL), with consistently better performance across all instances and an impressive average gap of 31.20%. Furthermore, compared to the First-Come-First-Served (FCFS) strategy, MAVNS consistently exhibits superior performance in all instances, highlighting a remarkable average gap of 98.67%. The research enables ports to better respond to unexpected situations and reduce the negative impacts caused by uncertainty. • A ship traffic scheduling problem under uncertainty is addressed using robust optimization. • A robust counterpart of the deterministic ship traffic scheduling model is derived. • A budget uncertainty is used to describe the arrival and departure times. • An efficient hybrid MAVNS algorithm that is provided. [ABSTRACT FROM AUTHOR]

Operational forecasting often requires predicting collections of related, multivariate time series data that are high-dimensional in nature. This can be tackled by fitting a single function to all series (global approach) or assuming each series as a separate prediction problem and fitting one function to each (local approach) – the global–local trade-off. Deep learning models inspired by different data generation processes aim to combine the benefits of global and local approaches. Specifically, these frequently propose feeding the statistical expressiveness of classical local models into more complex global networks. Following recent trends in neural networks, the theoretical foundations of these hybrid models can also explain the surge of Transformer-based time series forecasting applications, which showcase potential benefits for the global and local equilibrium. Dynamic reinforcement learning (RL) models have also been explored to optimize the balance of global and local signals in general prediction problems, frequently through Q-learning algorithms. RL models can be proposed to dynamically adjust the influence of global versus local information to improve predictive performance at both scales. This paper conducted a concise literature review focused on these two research streams to optimize the balance between globality and locality in forecasting collections of time series. It focuses on their evolution across time and hints at opportunities to close some of the research gaps by intersecting both propositions. We followed the Preferred Reporting Items for Systematic Reviews and Meta-analyzes (PRISMA) guidelines and achieved a selection of 143 publications since 2000. The main findings reveal that global models have achieved strong expressiveness in capturing the most complex structural patterns while still enabling probabilistic outcomes to be delivered through uncertainty estimates. On the other hand, RL based methods depict great benefits in mitigating the risks of generalization by imprinting contextual diversity when predicting each step ahead for each series. Within those, the adoption of other computational learning or evolutionary-based methods (e.g. Genetic Programming) to improve the parametrization of the learning policies is also highlighted as an area of future work yet to be uncovered. This review advances knowledge at the intersection of two distinct yet potentially complementary research areas, identifying opportunities to combine different methodological approaches in addressing the global–local trade-off in forecasting collections of time series. This is achieved by surfacing shared limitations in current research and presenting avenues for integrating distinct methodologies, namely by further developing the theoretical underpinnings of reinforcement learning techniques. With this work we seek to enhance understanding of the relevant research landscape and help inform future solutions by establishing a foundation for collaborative work. [Display omitted] • Review of global and local time series forecasting approaches including Deep Learning (DL) models combining global and local signals and Dynamic Reinforcement Learning (RL) methods optimizing this trade-off • Identification of opportunities to combine distinct forecasting methodologies like global DL models and RL based local modeling to address the global–local trade-off. • Search for ways to integrate RL techniques for time series forecasting by further developing their theoretical underpinnings based on limitations found in current research. • Main findings show global models achieve strong expressiveness capturing complex patterns while enabling probabilistic outcomes, and RL methods mitigate risks when predicting each step for each series [ABSTRACT FROM AUTHOR]

This paper addresses a crucial issue in the airline industry by tackling a dynamic pricing problem in the presence of patient customers, a scenario that has gained significance due to the revenue loss of airlines caused by customers' non-myopic decision-making. To effectively capture this non-myopic characteristic, we propose a Markov decision process (MDP) including a history of offered prices as a state variable. In contrast to previous studies, distributions of customers' properties are assumed to be unknown in advance for a more realistic representation of real-world scenarios. To deal with the new challenges of the problem, we propose utilizing a specific learning framework (i.e., deep exploration-based RL) that is unexplored in this domain. The numerical experiments demonstrate that its performance can be improved on the MDP we designed and show that it outperforms the benchmark algorithm. The structures of pricing policies generated from the bootstrapped deep Q-network algorithm imply that airlines should offer high and low prices alternately from the beginning of the sales period rather than increasing prices as time goes on. We also ascertain that more frequent consecutive high-priced periods can increase airlines' revenue in environments with higher customer patience levels. [ABSTRACT FROM AUTHOR]

Conventional Air Traffic Control is still predominantly being done by human Air Traffic Controllers, however, as the traffic density increases, the workload of the controllers increases as well. Especially for the area of unmanned aviation, driven by the rise in drones, having human controllers might become unfeasible. One of the methods that is currently being investigated for replacing the conflict resolution task of Air Traffic Control is Reinforcement Learning. As violation of the required separation margins, also called an intrusion, is an event of relatively low frequency, using Reinforcement Learning for this task comes with difficulties that can potentially be attributed to data imbalance. This paper artificially increased the traffic density during the training phase of the Reinforcement Learning method to investigate what the importance is of a balanced data set on the performance of the Reinforcement Learning method. It was found that as the traffic density increased, the Reinforcement Learning methods started to outperform the analytical methods. Beyond this it was found that methods trained at higher traffic densities, but tested at lower traffic densities, outperformed the methods trained at that specific density. This indicates that it might be better to always ensure that the training scenarios are more complex than anticipated during the execution phase, even if that results in unrealistic scenarios. [ABSTRACT FROM AUTHOR]

Wastewater treatment is important for maintaining a balanced urban ecosystem. To ensure the success of wastewater treatment, the tracking error between the crucial variable concentrations and the set point needs to be minimized as much as possible. Since the multiple biochemical reactions are involved, the wastewater treatment system is a nonlinear system with unknown dynamics. For this class of systems, this paper develops an online action dependent heuristic dynamic programming (ADHDP) algorithm combining the temporal difference with λ [TD(λ)], which is called ADHDP(λ). By introducing the TD(λ), the future n -step information is considered and the learning efficiency of the ADHDP algorithm is improved. We not only give the implementation process of the ADHDP(λ) algorithm based on neural networks, but also prove the stability of the algorithm under certain conditions. Finally, the effectiveness of the ADHDP(λ) algorithm is verified through two nonlinear systems, including a wastewater treatment system and a torsional pendulum system. Simulation results show that the ADHDP(λ) algorithm has higher learning efficiency compared to the general ADHDP algorithm. [ABSTRACT FROM AUTHOR]

The intermittent integration of renewable energy sources and the enhancement of energy-saving awareness on the demand side have posed significant challenges to energy management in smart grids. To address the supply–demand imbalance issues caused by these challenges, this paper proposes a novel multiple time-scale dispatch model. The model utilizes demand response strategy for demand side management (DSM) and employs dynamic pricing strategy for supply-side management. Furthermore, an advanced approach based on federated learning and Twin Delayed Deep Deterministic(TD3) policy gradient algorithm is proposed to construct a federated learning mechanism among multi agents. This approach performs distributed optimization on the multi-agent system to derive the optimal network weights for multiple stakeholders, resulting in the generation of an optimal dynamic pricing strategy that ensures the economic efficiency and secure operation of the smart grid. From the methodological perspective, this approach enhances privacy protection and decision-making autonomy for each stakeholder, while improving the optimization efficiency of the reinforcement learning (RL) algorithm. Based on simulation results, it is evident that the proposed method facilitates economically efficient transactions between supply and demand, and the improved distributed multi-agent federated reinforcement learning (MAFRL) approach holds promise as a feasible and optimal optimization method for economic dispatch in smart grid. [ABSTRACT FROM AUTHOR]

The rapid growth in the number of electric vehicles (EVs) has significantly increased the demand for electricity for residents. In addition, because the charging time of EVs highly coincides with the peak period of user electricity consumption, the disorderly charging of EVs will lead to the overload of the power grid transformer. Traditional control methods lack certain robustness and do not fully consider the uncertainty of EVs. As a result, the V2G participation rate of electric vehicles cannot be determined, and the control reliability is low. To solve the above problems, this paper designs a reinforcement learning framework of Long Short-Term Memory network and Improved Linear programming algorithm (LSTM-ILP) to control the V2G of EVs.This paper comprehensively considers the overall electric vehicle charging demand, discharge potential, large grid electricity price, aggregator, and users' interests demands. Firstly, aiming to minimize the charging and discharging fee of EVs and the load peak-to-valley difference of the power grid, a dynamic electricity price based on Long Short-Term Memory neural network (LSTM) is established. Then, the improved linear programming algorithm (ILP) is used to solve the charging and discharging optimization problem of EV, and the results are fed back to the input of the next iterative update of the LSTM, and finally, the optimal electricity price and EV charging and discharging schedule are achieved. The simulation results show that the LSTM-ILP framework can not only reduce the charging fee of electric vehicles, but also achieve the Peak and valley trimming of the grid load. Charging costs for EV users were reduced by 42.1% compared with unordered charging, and by 22% percent compared with orderly charging. [ABSTRACT FROM AUTHOR]

Considering the importance of the energy management strategy for hybrid electric vehicles, this paper is aiming at addressing the energy optimization control issue using reinforcement learning algorithms. Firstly, this paper establishes a hybrid electric vehicle power system model. Secondly, a hierarchical energy optimization control architecture based on networked information is designed, and a traffic signal timing model is used for vehicle target speed range planning in the upper system. More specifically, the optimal vehicle speed is optimized by a model predictive control algorithm. Thirdly, a mathematical model of vehicle speed variation in connected and unconnected states is established to analyze the effect of vehicle speed planning on fuel economy. Finally, three learning-based energy optimization control strategies, namely Q-learning, deep Q network (DQN), and deep deterministic policy gradient (DDPG) algorithms, are designed under the hierarchical energy optimization control architecture. It is shown that the Q-learning algorithm is able to optimize energy control; however, the agent will meet the "dimension disaster" once it faces a high-dimensional state space issue. Then, a DQN control strategy is introduced to address the problem. Due to the limitation of the discrete output of DQN, the DDPG algorithm is put forward to achieve continuous action control. In the simulation, the superiority of the DDPG algorithm over Q-learning and DQN algorithms in hybrid electric vehicles is illustrated in terms of its robustness and faster convergence for better energy management purposes. [ABSTRACT FROM AUTHOR]

Online ride-hailing has become one of the most important transportation ways. In the ride-hailing system, how to efficiently match orders with vehicles and dispatch idle vehicles are key issues. The ride-hailing platform needs to match orders with vehicles and dispatch idle vehicles efficiently to maximize social welfare. However, the matching and dispatching decisions at the current round may affect the supply and demand of ride-hailing in the future rounds since they will affect the future vehicle distributions in different geographical zones. In fact, vehicles in different zones at different times may have different values for the matching and dispatching results. In this paper, we use the vehicle value function to characterize the spatio-temporal value of vehicles in each zone and then use it to design the order matching and idle vehicle dispatching algorithm to improve the long-term social welfare. In addition, in the order matching, passengers may untruthfully report the maximum price they are willing to pay to maximize their own profits, which can affect the order matching and thus may result in the losses of the long-term social welfare. Therefore, we design a VCG based pricing algorithm to prevent the strategic behavior of passengers. We further run experiments to evaluate the proposed algorithm. The experimental results show that our algorithm can outperform the state-of-the-art algorithm in terms of social welfare by 11.7% and service ratio by 11.1%. This work can provide some useful insights for the online ride-hailing platform to design practical order matching and pricing strategies. [Display omitted] • We design a vehicle value function to optimize order matching and improve the long-term social welfare by using a reinforcement learning method. • We consider the dispatching of idle vehicles to a zone as a virtual order and process idle vehicle dispatching issue as (virtual) order matching as well. • We design a VCG based pricing method to prevent the strategic behavior of passengers and ensure positive profits for the platform. [ABSTRACT FROM AUTHOR]

Hyperspectral band selection plays a crucial role in reducing dimensionality, extracting relevant features, and improving computational efficiency in hyperspectral data analysis. Although numerous band selection methods have emerged in recent years, there remains a significant gap in exploring spatial structures and the diversity of ground objects. In this paper, we propose a region-wise latent feature fusion and graph filter embedded subspace clustering approach to address the band selection problem. Specifically, we segment the original hyperspectral image into diverse homogeneous regions using entropy rate superpixel segmentation. Next, we fuse the features from these regions into a consensus low-dimensional latent space, effectively capturing spatial information. To explore the spectral correlation among all bands, we employ a self-representation subspace clustering model on the fused latent features. Additionally, we apply a graph filter to the raw region-wise features to reduce redundant and noisy information present in the original data. By integrating these concerns into a unified framework, we facilitate mutual reinforcement among the learning sub-tasks. We conduct experiments on four public hyperspectral datasets, demonstrating a 1.93%, 0.50%, 0.76%, and 0.94% accuracy enhancement over current state-of-the-art methods. Additionally, our proposed optimization method enables the model to converge rapidly in under 15 iterations. [Display omitted] • We propose a novel approach for hyperspectral band selection. • Features from different regions are merged into a spatially-aware space. • Improves data quality by recover a smooth representation. • Demonstrates substantial accuracy boosts and rapid convergence. [ABSTRACT FROM AUTHOR]

Resource scheduling is an important part of military operation, especially in key-point air defense under saturation attack. Many achievements have been made in the field of radar resource scheduling and multi-aircraft scheduling. However, there is little research on the integrated scheduling of detection, tracking and attack, which will greatly increase the resource utilization to deal with the resource shortage problem caused by saturation attacks on key places. In this paper, we propose to autonomously accomplish real-time resource dispatching through end-to-end deep reinforcement learning (DRL), while allowing the commander's intervention to accomplish a variety of complex tactics. First, an integrated scheduling model of detection, tracking and interception is proposed and transformed into a sequential decision problem by introducing a disjunctive graph and a graph neural network (GNN) to extract node features. Subsequently, the Proximal Policy Optimization (PPO) algorithm is applied to learn the air defense environment (ADE), which is modeled as an Markov decision process (MDP). Benefitting from the powerful generalization capability of the policy network, our algorithm can adapt to scheduling missions of different sizes. Moreover, we propose a novel Human-Intelligence collaborative dynamic scheduling framework for emergency response. Simulation results indicate that our algorithm generates high-quality scheduling policies for defense resources, exhibiting superior performance than existing methods. In addition, the dynamic scheduling performance of the human and intelligence collaboration approach in response to multiple contingencies is proven. [ABSTRACT FROM AUTHOR]

A reliable energy management strategy (EMS) is of utmost importance in enhancing the fuel economy of fuel cell hybrid electric vehicles. However, correlation between fuel cell output power and temperature pose an urgent challenge of effectively integrating fuel cell thermal regulation with energy management. In this paper, an improved deep deterministic policy gradient (DDPG) EMS based on fixed time active disturbance rejection controller (FTADRC) is proposed. The proposed method not only achieves precise tracking of the optimal operating temperature in temperature regulation of fuel cell, but also seamlessly integrates thermal management with EMS. Specifically, through fuel cell thermal regulation experiments, the optimal operating temperature of the fuel cell is determined by fitting the relationship between temperature and fuel cell power. Then, a FTADRC temperature controller is designed to track the optimal operating temperature. Meanwhile, an integrated framework for EMS and fuel cell thermal regulation is developed, where the temperature difference and compensation power are considered in the DDPG state space and reward function. In order to accelerate the learning process of DDPG strategy, a temporal difference error network parameter adaptive update mechanism is proposed. Finally, simulations and experiments are conducted on three different driving cycles, where the obtained real vehicle test results show that compared with the improved DDPG strategy based on traditional thermal regulation, this method reduces hydrogen fuel consumption by 15.79% and improves average fuel cell efficiency by 1.87%. In addition, compared to traditional DDPG algorithms, the proposed DDPG algorithm reduces training time by 24%. [ABSTRACT FROM AUTHOR]

The dynamic job-shop scheduling problem (DJSP) is a type of scheduling tasks where rescheduling is performed when encountering the uncertainties such as the uncertain operation processing time. However, the current deep reinforcement learning (DRL) scheduling approaches are hard to train convergent scheduling policies as the problem scale increases, which is very important for rescheduling under uncertainty. In this paper, we propose a DRL scheduling method for DJSP based on the proximal policy optimization (PPO) with hybrid prioritized experience replay. The job shop scheduling problem is formulated as a sequential decision-making problem based on Markov Decision Process (MDP) where a novel state representation is designed based on the feasible solution matrix which depicts the scheduling order of a scheduling task, a set of paired priority dispatching rules (PDR) are used as the actions and a new intuitive reward function is established based on the machine idle time. Moreover, a new hybrid prioritized experience replay method for PPO is proposed to reduce the training time where samples with positive temporal-difference (TD) error are replayed. Static experiments on classic benchmark instances show that the make-span obtained by our scheduling agent has been reduced by 1.59% on average than the best known DRL methods. In addition, dynamic experiments demonstrate that the training time of the reused scheduling policy is reduced by 27% compared with the retrained policy when encountering uncertainties such as uncertain operation processing time. [ABSTRACT FROM AUTHOR]

With the development of mobile Internet and the popularity of public online consumption, the scale expansion of the e-commerce industry has driven the continuous growth of logistics business, but the increase in order volume has brought pressure and challenges to the end of logistics and distribution, and the emergence of crowdsourcing logistics It provides a new path for alleviating the current logistics and transportation dilemma. The Digital Twin (DT) can virtualize and learn the data of the physical space, and introduce DT into the crowdsourcing logistics. It can iteratively update the crowdsourcing logistics participant strategy by constructing a virtual space, so as to change the corresponding strategy in time. Considering the situation of crowdsourcing logistics workers signing contracts with platforms and colluding with platforms, this paper constructs a four-party evolutionary game model of temporary workers, contract workers, blockchain-based crowdsourcing platforms and task requester, and analyzes the evolution using replication dynamics method stabilization strategy. In the virtual scene of DT, multi-agent reinforcement learning is used to predict the evolution result of the current strategy, and a reward and punishment strategy is given to prevent workers from free-riding and platform false reporting. The simulation results show that the analyzed evolutionary stability strategy can make the crowdsourcing logistics system run continuously and healthily, and the participants can adjust the strategy correctly according to the prediction results of virtual crowdsourcing logistics. [ABSTRACT FROM AUTHOR]

Deep reinforcement learning (RL) is an advancing learning tool to handle robotics control problems. However, it typically suffers from sample efficiency and effectiveness. The emergence of Graph Neural Networks (GNNs) enables the integration of the RL and graph representation learning techniques. It realises outstanding training performance and transfer capability by forming controlling scenarios into the corresponding graph domain. Nevertheless, the existing approaches strongly depend on the artificial graph formation processes with intensive bias and cannot propagate messages discriminatively on explicit physical dependence, which leads to restricted flexibility, size transfer capability and suboptimal performance. This paper proposes a fuzzy attention mechanism-based heterogeneous graph neural network (FAM-HGNN) framework for resolving the control problem under the RL context. FAM emphasises the significant connections and weakening of the trivial connections in a fully connected graph, which mitigates the potential negative influence caused by the artificial graph formation process. HGNN obtains a higher level of relational inductive bias by conducting graph propagations on a masked graph. Experimental results show that our FAM-HGNN outperforms the multi-layer perceptron-based and the existing GNN-based RL approaches regarding training performance and size transfer capability. We also conducted an ablation study and sensitivity analysis to validate the efficacy of the proposed method further. [ABSTRACT FROM AUTHOR]

In this paper, an adversarial deep reinforcement learning-based control method is proposed to address the issue of robust depth tracking of an underactuated autonomous underwater vehicle in the presence of intrinsic coupled dynamics and external disturbances. First, long-short-term-memory neural network is presented to memorize and predict the changes in the state of vehicle, and a cascaded multilayer perception projects the output into action space of vehicle. Subsequently, adversarial deep reinforcement learning scheme is applied to the training of control agent by introducing an adversary which counteracts the control behavior, whereby the agent is enabled to learn the control strategy in different distributions of state transition. For evaluation of the performance, a control agent is pre-trained in simulation environment based on the reliable digital model of a real vehicle, and the simulation environment is paced at one iteration per second to align with real-time operations to ensure the portability of training result. Furthermore, the training cost is also extremely reduced. Finally, experiments are conducted with time-varying disturbances to further prove the feasibility of the proposed learning-based control scheme on a prototype of underwater vehicle in towing tank. Moreover, comparative experiment results show the better robustness performance of the learning-based control agent than that of classic line-of-sight based proportional–integral–derivative and adaptive line-of-sight based proportional–integral–derivative controllers in different scenarios. [ABSTRACT FROM AUTHOR]

An Automatic Voltage Regulator (AVR) system utilized to keep the terminal voltage of a synchronous generator at the desired level has received much attention among researchers. Designing an efficient and robust control scheme for the AVR system to maintain a specified voltage level is an important research area. From the control area perspective, reinforcement learning, an adaptive optimal control method, has received increasing attention in reference tracking problems. This article discusses a reinforcement learning approach to an AVR system and its experimental validation. A deep deterministic policy gradient (DDPG) agent working in continuous-time is designed offline to improve dynamic system characteristics of the AVR system besides its robustness against load disturbance, parameter uncertainties, and reference change. In the DDPG agent design process, the limits of the produced control signal are taken into account to perform a feasible simulation similar to a real-time application. The performance of the proposed learning-based controller is analyzed in three categories: transient and steady-state responses, stability analysis, and robustness analysis against parameter uncertainties, reference change, and load disturbance. A comparison with recently published papers employing Fuzzy-PID, PID-F, PI λ DND 2 N 2 , PIDD 2 ,and PID controllers in which various heuristic optimization algorithms were employed to optimally tune the controller parameters is made. Furthermore, to demonstrate that the behavior of the learning-based approach provides a stable and satisfactory performance, it is analyzed for a real synchronous generator connected to a 230 kV network using Matlab/Simulink environment. The results presented in this paper indicate that the proposed learning-based controller ensures the stability of the AVR system, significantly improves the regulating performance, and most impressively, is robust against parameter uncertainties, reference change, and load disturbance. [ABSTRACT FROM AUTHOR]

Given the continuous expansion of heating areas in recent years, the design of a precise and dependable district heating system (DHS) has become increasingly crucial. Traditional control decisions are made based on real-time environmental temperature feedback, often leading to uneven heating on the user side and affecting residents' comfort. This paper proposes an intelligent control strategy based on the deep reinforcement learning recurrent deterministic policy gradient (RDPG) algorithm for DHSs. To explore the control performance of the RDPG algorithm on DHS, we have meticulously modeled the pivotal components of the DHSs, namely plate heat exchangers, secondary heating pipe networks, and heat users. Moreover, taking into account the periodic factors in heating regulation, the traditional recurrent neural network (RNN) in the recurrent deterministic policy gradient (RDPG) algorithm has been replaced with the long short-term memory (LSTM) network. The proposed algorithm was trained using actual data from a heat exchange station in Tianjin and compared with reinforcement learning algorithms such as TD3, DPPO, DDPG, and A3C in terms of training rewards, effectiveness, and training stability. The results of the models are evaluated and visualized. Experimental results show that the proposed control method based on the RDPG algorithm, compared to other control schemes, can achieve the highest training reward and the most stable control performance, with an indoor temperature fluctuation range of only 0.1 °C. [ABSTRACT FROM AUTHOR]

In recent years, consortium blockchains have attracted considerable interest from the business communities and academia. To satisfy specific application requirements, appropriately configuring or adjusting the parameters is essential but challenging for developers when building the consortium blockchain. In this paper, the parameter configuration and adjustment for the consortium blockchain is transformed as a multi-objective optimization problem, and a novel explainable deep reinforcement learning (DRL) algorithm is proposed to solve the problem. On considering that existing DRL algorithms cannot be directly used in the consortium blockchain as they suffer from lacking of explainability, a causal model for configuring and adjusting the consortium blockchain's parameters is proposed and integrated into the DRL algorithm. The causal model can be used to derive causal explanations of the DRL algorithm to increase its trustworthiness. Furthermore, the causal model-based DRL (C-DRL) algorithm can perform causal inference before taking action to eliminate unreasonable exploration and improve the DRL algorithm's performance. The experimental results demonstrate the proposed algorithm provides the consortium blockchain with adaptive parameter configuration and adjustment to achieve sustainable high performance and security. Furthermore, the proposed algorithm increases the convergence speed of the vanilla algorithm by 49.3% and is more trustworthy than the vanilla DRL algorithm. • To achieve sustainable high performance and security, we transform the parameter configuration and adjustment for the consortium blockchain as a multi-objective optimization problem and provide a formal representation of the problem. • We propose a causal model of consortium blockchain's parameters using the structural causal model (SCM) formalism. • We propose a causal model-based DRL (C-DRL) algorithm to solve the multi-objective optimization problem. The C-DRL algorithm is equipped with the causal model of consortium blockchain's parameters. • The agent can use the causal model to derive the causal explanation for its action to provide trustworthiness. Furthermore, the agent with the causal model can utilize causal inference to eliminate unreasonable exploration and improve the convergence speed of the current DRL algorithm. • Extensive experiments are conducted to validate our proposal. The simulation results show that the agent running the C-DRL algorithm has more reasonable behavior than that running the model-free DRL algorithm. Moreover, the C-DRL algorithm converges faster than the model-free DRL algorithm. [ABSTRACT FROM AUTHOR]

Industrial cyber–physical systems (ICPS) are emerging platforms for various industrial applications. For instance, remote healthcare monitoring, real-time healthcare data generation, and many other applications have been integrated into the ICPS platform. These healthcare applications encompass workflow tasks, such as processing within hospitals, laboratory tests, and insurance companies for patient payments, which necessitate a sequential flow. The external wireless, fog, and cloud services within ICPS face security issues that impact end-users' healthcare applications. Blockchain technology offers an optimal solution for ICPS-enabled applications. However, blockchain technology for the ICPS platform is still vulnerable to cyberattacks, while microservices are essential for executing applications. This paper introduces the novel "Pattern-Proof Malware Validation" (PoPMV) algorithm designed for blockchain in ICPS. It exploits a deep learning model (LSTM) with reinforcement learning techniques to receive feedback and rewards in real-time. The primary objective is to mitigate security vulnerabilities, enhance processing speed, identify both familiar and unfamiliar attacks, and optimize the functionality of ICPS. Simulations demonstrate the superiority of the proposed approach compared to current blockchain frameworks, showcasing dynamic allocation of microservices and improved security with comprehensive attack detection by 30%. [ABSTRACT FROM AUTHOR]

This paper presents a novel method for multi-objective optimization of truss design utilizing multi-agent reinforcement learning and graph representation. The agents are trained to modify solutions to increase the hypervolume and spread the distribution of the non-dominated solutions. This iterative modification is modeled as a Markov game where multiple agents interact with the environment by changing the state of the environment, i.e., the solution and the current non-dominated solutions. Agents observe data representations of the state, modify each solution in the current non-dominated solutions, and receive their reward based on the improvement of the current non-dominated solutions. Each agent is modeled by the multi-agent deep deterministic gradient consisting of a policy function that predicts actions from its observation data and a value function that estimates rewards from the observation data, its action, and the actions of the other agents. The non-dominated solutions are represented as graph data and observed by the agents as the observation data through graph representation. The proposed method is applied to three multi-objective optimization problems: (1) a simple mathematical problem, (2) a 10-bar truss problem to minimize the structural volume and the displacement, and (3) trade-off designs of trusses to minimize the structural volume and bring the shape closer to the target shape. These numerical examples show the versatility of the trained agents to obtain superior solutions in practical structural optimization problems, compared to a conventional method. • A multi-objective optimization method of truss using multi-agent reinforcement learning is proposed. • Agents designed to observe solutions and Pareto front and improve the current non-dominated solutions. • The method can obtain Pareto fronts using fewer computations when compared to a conventional method. [ABSTRACT FROM AUTHOR]

The Newton-Raphson-Based Optimizer (NRBO), a new metaheuristic algorithm, is suggested and developed in this paper. The NRBO is inspired by Newton-Raphson's approach, and it explores the entire search process using two rules: the Newton-Raphson Search Rule (NRSR) and the Trap Avoidance Operator (TAO) and a few groups of matrices to explore the best results further. The NRSR uses a Newton-Raphson method to improve the exploration ability of NRBO and increase the convergence rate to reach improved search space positions. The TAO helps the NRBO to avoid the local optima trap. The performance of NRBO was assessed using 64 benchmark numerical functions, including 23 standard benchmarks, 29 CEC2017 constrained benchmarks, and 12 CEC2022 benchmarks. In addition, the NRBO was employed to optimize 12 CEC2020 real-world constrained engineering optimization problems. The proposed NRBO was compared to seven state-of-the-art optimization algorithms, and the findings showed that the NRBO produced promising results due to its features, such as high exploration and exploitation balance, high convergence rate, and effective avoidance of local optima capabilities. In addition, the NRBO also validated on challenging wireless communication problem called the internet of vehicle problem, and the NRBO was able to find the optimal path for data transmission. Also, the performance of NRBO in training the deep reinforcement learning agents is also studied by considering the mountain car problem. The obtained results also proved the NRBO's excellent performance in handling challenging real-world engineering problems. [ABSTRACT FROM AUTHOR]

High-dimension inputs limit the sample efficiency of deep reinforcement learning, increasing the difficulty of applying it to real-world continuous control tasks, especially in uncertain environments. A good state embedding is crucial for the performance of agents on downstream tasks. The bisimulation metric is an excellent representation learning method that can abstract task-relevant invariant latent embeddings of states based on behavior similarity. However, since only one time-step transition is considered, the features captured by this method we call short-term dynamics. We think long-term dynamics are also important for state representation learning. In this paper, we present Invariant R epresentations L earning with F uture Dynamics (RLF) , which uses graph neural networks to learn long-term dynamics and trains the representation network based on a new state metric inspired by bisimulation relation. We experimented with our method on continuous control tasks from DeepMind Control Suite and showed that the RLF can mine more stable embeddings than the state-of-the-art representation learning methods for both state and pixel inputs. The learned policy on top of the embeddings has higher sample efficiency and performance and generalizes well in different tasks. [ABSTRACT FROM AUTHOR]

Target-Driven Semantic Navigation (TDSN) shows great potential to be applied in intelligent domestic assistants supporting humans with daily activities. Although numerous methods have been explored to achieve efficient static TDSN, socially aware TDSN in dynamic and crowded scenarios remains challenging and has not been adequately investigated. The main challenges come from the complex human–robot interaction mechanisms and the semantic relation exploitation, which requires the robot to understand the surroundings and perform foresighted behaviors. In this paper, a TDSN strategy named SemNav-HRO is proposed by considering Human–Robot–Object (HRO) ternary feature fusion. Specifically, a Deep Reinforcement Learning (DRL) based Dual-Channel Value Estimation Network (DCVEN) is first proposed by integrating multi-granularity map features and social awareness to learn crowded TDSN strategies. Meanwhile, the tricky and socially aware TDSN problem is slackened by eliminating the dependence on costly features (e.g., pedestrian speed) and introducing a pedestrian trajectory predictor. For the learning and evaluation of crowded TDSN strategies, a novel and semantic-rich simulator with complex layouts is constructed based on realistic domestic scenes, instead of employing the previous naive simulation settings. Experimental results show that our method relatively improves the navigation success rates by 12. 8 % ∼ 25. 5 % and 14. 6 % ∼ 19. 2 % compared to the baselines on the MP3D and Gibson datasets, respectively. Furthermore, we experimentally verify the promising generalization and interpretability of our method. • This is the first work to study target-driven semantic navigation in crowded scenes. • A novel spatio-temporal aware pedestrian trajectory predictor is designed. • A novel multi-granularity semantic relationship modeling technique is proposed. • A dual-channel value estimation network is proposed for HRO ternary feature fusion. • A novel and semantic-rich simulator with complex layouts is constructed. [ABSTRACT FROM AUTHOR]

The unmanned lunar rover is essential for lunar exploration and construction. Executing environment differ from what humans get since communication needs time from Earth to Moon. Considering possible discrepancies between the pre-considered environment from the planner and the real environment for sampling tasks on the moon, a planner that generates short plans quickly should be used. Therefore, a planner for both standard and emergency planning based on deep reinforcement learning (DRL) is demonstrated in this paper. This planner can create a full-range plan 13.5 times faster than the traditional planner on complex problems or 10.1 times faster while controlling the rover step-by-step in the state space. Based on a specific moon sampling scenario, we propose a tracking reward guiding the rover searching in the states in the deep reinforcement learning architecture which is presented and created by a state space representation by matrix, randomly available training state pairs and the plans generated by a custom breadth-first search (BFS) planner for the tracking reward. The BFS planner obtains a custom state hash algorithm and a preparation to train state pairs for safety and flexibility. Tests on training and planning are performed to validate the effectiveness, robustness and customization of the proposed method in a planning domain with multiple rovers. Our model can handle three kinds of emergencies, even if they occur frequently. The success rate is beyond the state-of-the-art model. While facing emergencies, the average response time of our model is 324 times faster than the classical planner. • An architecture of deep reinforcement learning for automated planning is utilized on lunar rovers. • Matrix representation of state space, the way of checking preconditions and applying actions' influence are designed. • An algorithm for generating random valid state pairs by customized BFS and state hash algorithm for planning is proposed. • A strategy of arranging training batches to ensure safety and adaptability is demonstrated. • A progressive planning scenario is proposed to validate the performance and effectiveness of the planner. [ABSTRACT FROM AUTHOR]

This paper addresses how physical knowledge can improve machine learning in process control. A data-driven tracking control framework using physics-informed neural networks (PINNs) and deep reinforcement learning (DRL) is proposed for dynamical systems, which is particularly important when iterations or repetitions of data collection experiments are limited or even not allowed. An indirect control approach is followed to accomplish this. First, a new process model is identified using PINNs trained from a model representing the experimental data collected from the plant. Then, the DRL-based control agent utilizes the identified PINN model to deliver an offline policy. We evaluate our framework for data-driven tracking control of nonisothermal CSTR, considering measurement noise and stochastic dynamics for closed-loop control. Our approach performed comparably to an NMPC without requiring a model to predict the process dynamics. • Physical knowledge was implemented to improve machine learning. • A indirect data-driven control approach was developed. • Physics-informed neural networks and deep reinforcement learning are combined. • The developed framework was tested against nonlinear model predictive control. [ABSTRACT FROM AUTHOR]

Astronomical seeing is the primary indicator of the observation quality of large-scale optical telescopes. Forecasting astronomical seeing is an essential and challenging task in the operation of large optical telescopes, as it can help observers more efficiently arrange observation tasks as well as providing hints for mechanism optimization of optical telescopes. In this paper, we propose an efficient data-driven astronomical seeing forecasting method by leveraging the environment monitoring data from multiple sensors deployed with optical telescopes. The basic idea is to use a large number of random yet fixed convolution kernels to extract features from the original data, and then train a simple tree-based predictor based on these features. In particular, the raw monitoring data is transformed by differential operations, enabling a more explicit representation of spatial and temporal features; then an elaborate convolution mechanism, including the convolution kernel and the pooling operator, is designed to extract a variety set of features of the input time series, which serve as the foundation for constructing the time series forecasting model. In contrast to existing deep learning models, there is no need to calculate and back-propagate the gradients during the training procedure, making the proposed method much more efficient. We collect real-world monitoring data from Large Sky Area Multi-Object Fiber Spectroscopy Telescope (LAMOST), one of the world's largest optical reflecting telescopes, and conduct extensive empirical evaluations. The experimental results demonstrate that the proposed method can achieve state-of-the-art prediction accuracy, i.e., 0.1125 in MAE, 0.0252 in MSE, and 0.0322 in MAPE, outperforming most deep-learning-based forecasting models. Moreover, it surpasses both machine learning and deep learning approaches in terms of training efficiency with a significant margin. [Display omitted] • The task of forecasting astronomical seeing for optical telescopes is investigated. • A variety of real-world monitoring data of LAMOST is collected. • A new multivariate time series forecasting method — SeeROCKET is proposed. • SeeROCKET achieves very promising performance both in accuracy and efficiency. [ABSTRACT FROM AUTHOR]

Aeroengine fault detection is an important means to ensure flight safety. The application premise of data driven fault detection method is that all data come from the same distribution. However, this assumption is invalid in the actual engine fault detection, because the engine state will change with the increase of operating time, and the collected data will also have distribution differences. Due to the high cost of collecting engine data, it is difficult to collect enough data from the current state. Fortunately, transfer learning can transfer data information from other fields to the target field, thus alleviating the problem of data scarcity in the target domain. Therefore, from the idea of transfer learning, this paper proposes a cross domain aeroengine fault detection method, viz. transfer learning based on kernel perception algorithm with uneven margins (TL-KPAUM). The proposed method is divided into two stages. In the first stage, KPAUM is trained using the data in the source domain to extract the information in the source domain. In the second stage, the data in the target domain is used to realize the adaptation to the target domain. Compared with several baseline methods, TL-KPAUM has better performance when the amount of data in the target domain is small. Finally, using the simulation data of turbofan engine, the fault detection experiments of degraded state are designed, and the results show that this method is effective. [Display omitted] • A transfer learning algorithm is proposed for aeroengine fault detection. • The proposed method is applied to engine fault detection in different domains. • Simulation results show the effectiveness of this method compared with other methods. • The proposed method has good performance in the case of sparse data in the target domain. [ABSTRACT FROM AUTHOR]

The collaborative optimization dispatching of multiple energy flows plays a crucial role in achieving the economic and efficient low-carbon operation of integrated energy systems (IESs). However, the dispatching problem for IESs is characterized by high dimensionality, non-linearity, and complex coupling. Furthermore, the integration of renewable energy sources and flexible loads has transformed the IES into a complex dynamic system with significant uncertainty. Traditional intelligent optimization algorithms exhibit poor adaptability and lengthy solution computation time when tackling such problems. In contrast, deep reinforcement learning (DRL), as an interactive trial-and-error learning method, has shown improved decision-making capabilities. In view of this, a data-driven soft actor-critic (SAC) deep reinforcement learning-based approach is proposed in this paper for interval optimal dispatch of IESs considering multiple uncertainties. First, the basic principle of SAC reinforcement learning is introduced in detail, and the basic framework of reinforcement learning for interval optimal scheduling of IESs is constructed. Then, the environment model of agent interaction is constructed, and the action and state space of DRL, as well as the reward mechanism and neural network structure, are designed. Finally, a typical IES case is experimentally analyzed and compared with three popular DRL algorithms and five state-of-the-art intelligent optimization algorithms. The experimental results demonstrate the advantages and effectiveness of the proposed method in solving the optimal dispatching of IESs. [ABSTRACT FROM AUTHOR]

We explore the reinforcement learning approach to designing controllers by extensively discussing the case of a quadcopter attitude controller. We provide all details allowing to reproduce our approach, starting with a model of the dynamics of a crazyflie 2.0 under various nominal and non-nominal conditions, including partial motor failures and wind gusts. We develop a robust form of a signal temporal logic to quantitatively evaluate the vehicle's behavior and measure the performance of controllers. The paper thoroughly describes the choices in training algorithms, neural net architecture, hyperparameters, observation space in view of the different performance metrics we have introduced. We discuss the robustness of the obtained controllers, both to partial loss of power for one rotor and to wind gusts and finish by drawing conclusions on practical controller design by reinforcement learning. [ABSTRACT FROM AUTHOR]