Search

Natural gas is an emerging and reliable energy source in transition to a low-carbon economy. The natural gas transportation pipeline network systems are crucial when transporting natural gas from the production endpoints to processing or consuming endpoints. Optimizing the operational efficiency of compressor stations within pipeline networks is an effective way to reduce energy consumption and carbon emissions during transportation. This paper proposes an optimization framework for natural gas transportation pipeline networks based on deep reinforcement learning (DRL). The mathematical simulation model is derived from mass balance, hydrodynamics principles of gas flow, and compressor characteristics. The optimization control problem in steady state is formulated into a one-step Markov decision process (MDP) and solved by DRL. The decision variables are selected as the discharge ratio of each compressor. By the comprehensive comparison with dynamic programming (DP) and genetic algorithm (GA) in three typical element topologies (a linear topology with gun-barrel structure, a linear topology with branch structure, and a tree topology), the proposed method can obtain 4.60% lower power consumption than GA, and the time consumption is reduced by 97.5% compared with DP. The proposed framework could be further utilized for future large-scale network optimization practices.

As the energy landscape evolves towards sustainability, the accelerating integration of distributed energy resources poses challenges to the operability and reliability of the electricity grid. One significant aspect of this issue is the notable increase in net load variability at the grid edge.Transactive energy, implemented through local energy markets, has recently garnered attention as a promising solution to address the grid challenges in the form of decentralized, indirect demand response on a community level. Model-free control approaches, such as deep reinforcement learning (DRL), show promise for the decentralized automation of participation within this context. Existing studies at the intersection of transactive energy and model-free control primarily focus on socioeconomic and self-consumption metrics, overlooking the crucial goal of reducing community-level net load variability.This study addresses this gap by training a set of deep reinforcement learning agents to automate end-user participation in an economy-driven, autonomous local energy market (ALEX). In this setting, agents do not share information and only prioritize individual bill optimization. The study unveils a clear correlation between bill reduction and reduced net load variability. The impact on net load variability is assessed over various time horizons using metrics such as ramping rate, daily and monthly load factor, as well as daily average and total peak export and import on an open-source dataset.To examine the performance of the proposed DRL method, its agents are benchmarked against a near-optimal dynamic programming method, using a no-control scenario as the baseline. The dynamic programming benchmark reduces average daily import, export, and peak demand by 22.05%, 83.92%, and 24.09%, respectively. The RL agents demonstrate comparable or superior performance, with improvements of 21.93%, 84.46%, and 27.02% on these metrics. This demonstrates that DRL can be effectively employed for such tasks, as they are inherently scalable with near-optimal performance in decentralized grid management.

Buildings are responsible for 30 % oof the world’s energy consumption, about half of which is consumed by Heating, Ventilation, and Air Conditioning (HVAC) systems. Intelligent control of these can significantly reduce global energy consumption. However, these systems are also one the most important means of providing comfort to occupants within buildings. These two control objectives may conflict with each other, which means that reducing the energy consumption of the HVAC systems may lead to dissatisfaction among buildings’ occupants. Thus, in most cases, to handle the trade-off, weights are assigned to these objectives which indicate their importance. The priority of these goals may change throughout the operation, meaning that the value of assigned weights must change. To solve this problem, existing methods needed to be retrained every time that the weights changed, which is time-consuming and costly. Thus, an algorithm that can adapt to weight changes in operation time is needed. In this research, a dynamic multi-objective deep reinforcement learning algorithm is introduced to control the power of the HVAC system. Since it is needed to control the continuous power of the HVAC system, a Deep Deterministic Policy Gradient (DDPG) algorithm empowered by Tunable Training, is proposed which can adapt to changes of objectives weights. The controller can adapt to changes in the importance of objectives during operation and does not need to be retrained. The proposed method is compared with a dynamic multi-objective Deep Q-Network (DQN) Algorithm, which is used for discrete control, in a smart home. The results show that the cost of energy in the proposed method is up to 14 % lower than the DML-DQN algorithm.

Trucks consume a lot of energy. Hybrid technology maintains a long range while realizing energy savings. Hybrid is therefore an effective energy-saving technology for trucks. Recovery of engine waste heat through the organic Rankine cycle further enhances engine efficiency and provides effective thermal management. However, the powertrain greatly increases the complexity of energy management system. In order to design an energy management system with high efficiency and robustness, this study proposes a deep reinforcement learning embedded rule-based energy management system. This method optimises the key parameters of the rule-based energy management system by inserting deep reinforcement learning into it. Therefore, this scheme combines the good optimization effect of deep reinforcement learning and the excellent robustness of rule. In order to verify the feasibility of this scheme, this study builds the system dynamic model and carries out a simulation study. Subsequently, a hybrid powertrain semi physical experimental bench was constructed and a rapid control prototype experimental study was carried out. The simulation results show that the deep reinforcement learning embedded rule-based energy management system can reduce the energy consumption by 4.31 % compared with the rule-based energy management system under the C-WTVC driving cycle. In addition, energy saving and safe operation can also be achieved under other unfamiliar untrained driving cycles. The rapid control prototype experimental study shows that the deep reinforcement learning embedded rule-based energy management system has good agreement in experiment and simulation, which demonstrates the potential for real vehicle engineering applications and promotes the engineering application of deep reinforcement learning.

In this paper, a dual deep Q-network (DDQN) energy management model based on long-short memory neural network (LSTM) speed prediction is proposed under the model predictive control (MPC) framework. The initial learning rate and neuron dropout probability of the LSTM speed prediction model are optimized by the genetic algorithm (GA). The prediction results show that the root-mean-square error of the GA-LSTM speed prediction method is smaller than the SVR method in different speed prediction horizons. The predicted demand power, the state of charge (SOC), and the demand power at the current moment are used as the state input of the agent, and the real-time control of the control strategy is realized by the MPC method. The simulation results show that the proposed control strategy reduces the equivalent fuel consumption by 0.0354 kg compared with DDQN, 0.8439 kg compared with ECMS, and 0.742 kg compared with the power-following control strategy. The difference between the proposed control strategy and the dynamic planning control strategy is only 0.0048 kg, 0.193%, while the SOC of the power battery remains stable. Finally, the hardware-in-the-loop simulation verifies that the proposed control strategy has good real-time performance.

The Space-Air-Ground Integrated Network (SAGIN) offers a promising solution for seamless connectivity, high data rates, and wide-area coverage. However, its multi-segment architecture poses significant challenges in efficient resource management and Quality of Service assurance across diverse services. To address these challenges, we propose an Intent-Based Networking (IBN) system that streamlines network automation within the SAGIN ecosystem. Our system model integrates an IBN module with the SAGIN infrastructure, allowing network operators to express their service intents, such as Low-Latency Virtual Network Requests (LLVNRs) and High-Bandwidth Virtual Network Requests (HBVNRs), along with their respective QoS requirements. To tackle the inherent complexity, we employ a Deep Deterministic Policy Gradient (DDPG) based DRL-IBN framework. The DRL agent interacts with the SAGIN environment using a feature matrix extracted via Graph Neural Network (GNN), facilitating informed decision-making for resource allocation based on VNR acceptance. Through extensive simulations and numerical evaluations, we demonstrate the superiority of our proposed DRL-IBN algorithm over baseline approaches, such as LC-VNE and RW-MM-SP, in maximizing system utility, ensuring QoS satisfaction, and enabling efficient resource utilization in the dynamic and heterogeneous SAGIN environment. Our results indicate that the DRL-IBN framework achieves higher VNR acceptance ratios, better resource utilization, and more effective QoS assurance based on minimum QoS violation, proving its effectiveness and robustness in managing the complexities of the SAGIN network.

Netted radar system is used to implement detection of aerial targets to cover high-value military sites. To address the lack of an effective strategy for Unmanned Aerial Vehicles (UAVs) to perform Multi-Target Reconnaissance (MTR) against netted radar system, a reconnaissance pseudo-target is shaped and used to guide the UAV to quickly approach a key reconnaissance area. By grouping radar positions, solving for multiple pseudo-targets, and integrating these pseudo-targets, we ultimately obtain an invisible pseudo-target that spans the entire radar detection range. The use of the pseudo-target reduces the dimensionality of the UAV’s observation state, thereby accelerating algorithm convergence. In addition, to solve the problem of the difficulty in determining the reconnaissance importance weights of multiple radars, a state-reward shaping equation combining the pseudo-target and real targets is designed. Finally, a deep reinforcement learning framework based on partially observable Markov processes is built, while a modular scenario pre-training method is used to improve the convergence of the policy network in the case of sparsely distributed high-quality samples. The experimental results show that the trained UAVs can successfully perform reconnaissance on up to six radar targets under physical constraints, and the completion rate for the MTR missions of six radars can reach 73 %.

Deep reinforcement learning (DRL) can significantly improve the autonomy and effectiveness of air combat maneuver decision (ACMD). The design of reward functions faces significant challenges due to the long duration and large state space of air combat. Most existing methods rely on subjective judgment and cumbersome parameter tuning. Therefore, this paper proposed a reward design paradigm that provides practical guidance for the agent to win within visual range (WVR) air combat. Firstly, we proposed a dynamic situation assessor(DSA) based on the improved entropy weight method, which can improve the accuracy, contrast, and objectivity of air combat situation evaluation. In addition, we design a parameter optimizer(PO) based on the improved sparrow search algorithm (SSA) to achieve the automation of parameter tuning for proximal policy optimization (PPO), and applied it in the reward optimization, which improves the training efficiency significantly. Finally, the effectiveness of our work is evaluated through comparative experiments with state-of-the-art (SOTA) methods on a high-fidelity air combat simulation platform (JSBSim). The training process and win rate tests confirm the effectiveness of our reward design paradigm and show good training efficiency compared to existing methods.

Robots are increasingly operating in highly complex and dynamic scenarios where they must continuously perceive their environment, learn from new experiences, and apply acquired knowledge to complete their tasks effectively. In these environments, the potential situations a robot encounters can become too vast to handle with predefined conditions. As a result, autonomous robots must incorporate learning methods that accurately represent the environment, make informed decisions, and optimize learning speed, task performance, and computational resources. Given the recent advancements of Deep Reinforcement Learning over classical Reinforcement Learning, this paper presents a Deep Reinforcement Learning system for biologically inspired, socially-driven decision-making in autonomous robots operating in such intricate environments with countless variations. This work formulates a learning framework as a Markov Decision Process, enabling robots to demonstrate adaptive social behaviour by integrating internal and external factors. The robot’s state includes 11 variables derived from the robot’s motivations, user perception, ambient light, and social norms, allowing the robot to select from ten possible actions autonomously. This study aims to develop fully autonomous robots that operate autonomously, learning and adapting to complex environments while maintaining an optimal balance between the robot’s internal and social well-being. We compare eight state-of-the-art DRL algorithms to identify the best-performing approach and implement the learning system into our Mini social robot. The results highlight Rainbow as the most effective solution, enabling the Mini robot to exhibit highly adaptive, autonomous behaviour in challenging social environments. These results allow autonomous robots to increase their capabilities and reduce human supervision.

In recent years, addressing the inherent uncertainties within Combinatorial Optimization Problems (COPs) reveals the limitations of traditional optimization methods. Although these methods are often effective in deterministic settings, they may lack flexibility and adaptability to navigate the uncertain nature of real-world COP/s. Deep Reinforcement Learning (DRL) has emerged as a promising approach for dynamic decision-making within these complex environments. Yet, the application of DRL in solving COP/s highlights key limitations for the generalization process across various problem instances without extensive retraining and customization for each new variant, leading to notable computational costs and inefficiencies. To address these challenges, this paper introduces a novel framework that combines the adaptability and learning capabilities of DRL with the efficiency of Transfer Learning (TL) and Neural Architecture Search. This framework enables the leveraging of knowledge gained from solving COP/s to enhance the solving of different but related COP/s, thereby eliminating the necessity for retraining models from scratch for each new problem variant to be solved. The framework was evaluated on over 1,500 benchmark instances across 10 stochastic and deterministic variants of the vehicle routing problem. Across extensive experiments, the approach consistently improves solution quality and computational efficiency. On average, it achieves at least a 5% improvement in solution quality and a 20% reduction in CPU time compared to state-of-the-art methods, with some variants showing even more substantial gains. For large-scale instances over 200 customers, the TL process requires only 10-15% of the time needed to train models from scratch, while maintaining solution quality, laying the groundwork for future research in this area.

In autonomous air combat, tactics are inherently complex, and control inputs are continuous. Traditional reinforcement learning (RL) algorithms often rely on discretization or independent Gaussian assumptions, which fail to capture correlations between control variables, limiting the expressiveness of strategies. Moreover, the highly dynamic and complex nature of battlefield scenarios poses significant challenges for conventional neural networks in modeling the long-term evolution of sequential data. To address these challenges, this paper proposes a novel algorithm, T-MGMM, which integrates Transformer networks with a Multivariate Gaussian Mixture Model (MGMM). The self-attention mechanism of Transformers effectively captures dependencies between variables and key situational information. Meanwhile, MGMM utilizes non-diagonal covariance matrices to account for correlations between actions, enhancing action modeling. This synergy ensures precise sequence modeling and flexible decision-making, making T-MGMM particularly well-suited for the complexities of air combat scenarios. To further improve optimization stability, we introduce internal Kullback-Leibler divergence regularization. Experimental results demonstrate that T-MGMM outperforms state-of-the-art algorithms, achieving higher Elo scores within the same training steps, and showcasing superior effectiveness and robustness in air combat decision-making.

The multiobjective traveling salesman problem (MOTSP) is a class of classical multiobjective combinatorial optimization problems (MOCOPs), attracting significant interest across various disciplines. Due to the NP-Hard nature, traditional algorithms struggle to find feasible solutions in a short period of time. To this end, this paper proposes a deep reinforcement learning algorithm based on graph weight multi-pointer network (GWMPN), which only needs to perform forward propagation on the GWMPN model to obtain the solution of MOTSP, thus significantly improving the solution efficiency. Specifically, we encode the problem instance and weight vector by means of the graph weight encoder of the GWMPN model to construct the overall encoded feature with enhanced representation capabilities. Subsequently, the encoded feature as the context vector is fed into the multi-pointer decoder to construct solutions through diversity interactions. Experimental results demonstrate that our algorithm solves the MOTSP more rapidly than classical heuristic algorithms and exhibits superior performance and generalization compared with learning-based algorithms within a reasonable time frame.

The diverse applications of Machine-Type Communication (MTC) lead to exponential growth in Machine to Machine traffic. In connection with MTC deployment, a large number of devices are expected to access the wireless network simultaneously, resulting in network congestion. The conventional Random Access mechanism lacks the capability to handle the large number of access attempts expected from massive MTC (mMTC). Additionally, mMTC transceivers often operate by generating short data packets in a sporadic and sometimes unpredictable manner. To address the growing need for efficient communication in massive machine-type communication scenarios we propose an innovative solution, called Deep Reinforcement Learning-Based Multi-Access (DRLMA). Our model considers the Base Station (BS) as an agent navigating the landscape of machine-type communication devices. This agent dynamically switches between grant-based and grant-free access to leverage their strengths. We address the multi-access problem, formulating it as a Partially Observable Markov Decision Process (POMDP), to better understand and tackle challenges associated with dynamic access policies. Leveraging Deep Reinforcement Learning techniques, our approach optimizes sporadic traffic patterns, crafting an adaptable access policy to maximize both network throughput and energy efficiency under the battery constraint. Simulation results show that proposed DRLMA scheme outperforms traditional access schemes and existing access protocols in sporadic traffic in terms of the energy efficiency, throughput and network life time.

The counting of dense targets plays a crucial role in public safety, intelligent traffic management, and other fields, providing critical data support for multiple industries. However, existing counting methods are mostly suitable for visually specific categories and scenes with uniformly distributed targets. When facing complex scenarios with large fluctuations in target density and high freedom of movement, in reality, the effectiveness of these methods is greatly compromised. To address this issue, this study proposes RegionNet: a multi-object counting model based on intelligent region partitioning. Compared to existing methods, we have drawn inspiration from deep reinforcement learning principles to design a region selection module. This module optimizes the ranges of dense and sparse areas in images effectively and reasonably. Additionally, by introducing a dense region counting module and a sparse region counting module, which utilize density estimation and object detection techniques respectively, the model effectively addresses the counting tasks in different regions. This approach successfully overcomes the issue of reduced accuracy faced by traditional counting methods when dealing with significant variations in target distribution scales, thereby significantly enhancing the counting accuracy across various regions. Experimental results demonstrate that the method proposed in this paper significantly outperforms existing technologies in terms of counting performance on multi-visual category target datasets. It exhibits higher accuracy and exceptional robustness.

Reinforcement learning in environments with visual observations presents challenges due to incomplete individual observations. The lack of complete information leads to increased uncertainty in decision-making, which requires agents to be supplemented with a memory module to retain information about previous observations. Our paper proposes a novel spatial memory mechanism with a flexible access system based on the multihead attention mechanism. Through experiments in the Atari benchmark and multiple autonomous driving environments, our approach outperforms agents using classical convolutional and recurrent neural networks. Further analysis reveals repeated interpretive patterns in attention distribution among trained agents. This study highlights the effectiveness of spatial memory and attention mechanisms in improving the efficiency of deep reinforcement learning in partially observable environments.

Quadratic Unconstrained Binary Optimization (QUBO) is a versatile approach used to represent a wide range of NP-hard Combinatorial Optimization (CO) problems through binary variables. The transformation of QUBO to an Ising Hamiltonian is recognized as an effective method for solving key optimization problems using quantum algorithms. Recently, PI-GNN, a generic framework, has been proposed to address CO problems over graphs based on QUBO with Hamiltonian loss function to train the underlying GNN architecture. Though PI-GNN is highly scalable, it exhibits a noticeable decrease in terms of the number of satisfied constraints with higher graph densities. In this paper, firstly, we identify the limitations and empirically present our strategy to improve PI-GNN’s performance. Secondly, we formulate and evaluate two strategies to integrate QUBO-Hamiltonian as the generic loss function in Reinforcement learning-based (RL) frameworks. The major contribution of our work lies in understanding the feasibility and quality of the QUBO-based generic reward function in an unsupervised RL setup in addressing graph-based CO problems. Empirically, through our empirical evaluation (Our implementation can be found in https://tinyurl.com/5apnymz7), we have observed up to 44% improvement in terms of the number of satisfied constraints over PI-GNN in a representative Max-Cut problem.

In recent years, mobile robots have been increasingly used in a variety of industries, such as manufacturing, logistics, health care, and security services. Effective autonomous navigation preventing collisions for mobile robots has become crucial for efficiently performing tasks in these diverse applications. Deep reinforcement learning (DRL) is promising for path planning and autonomous navigation, but Deep Q-Network is limited to discrete actions. To extend to continuous action space, the deep deterministic policy gradient (DDPG) algorithm was developed. However, it suffers from insufficient exploration and overestimation. To solve these challenges, this paper proposes the adaptive parameter space noise with parallel experience replay, double critics and double actors DDPG (AP-D4PG) algorithm, an enhanced variant of DDPG. It incorporates adaptive parameter space noise in actor networks for better exploration, double critics and double actors to mitigate overestimation and improve the estimate accuracy of target Q value, and parallel experience replay to make a better balance between exploration and exploitation, thereby significantly improving the performance of autonomous navigation for mobile robots. Furthermore, a novel and powerful target-oriented reward function is employed to boost learning efficiency. Additionally, to further bolster the stability and robustness, useful improvements including gradient clipping mechanism and Xavier initialization are employed. Subsequently, the efficacy of the proposed model is assessed through numerical experiments with the robot operating system (ROS) and Gazebo. The experimental findings demonstrate that the AP-D4PG algorithm achieves faster convergence and enhanced robustness, with a 3.83% and 7.78% increase in navigation accuracy, and a 61.14% and 48.93% higher average score in static and dynamic scenarios, respectively, compared to traditional DDPG. These improvements and the robustness of the AP-D4PG algorithm in both static and dynamic environments indicate that refining network architectures, exploration strategies, and reward mechanisms can enhance the performance of DRL algorithms. This opens up new possibilities for improving the efficiency and effectiveness of other DRL methods.

Deep reinforcement learning (DRL) has achieved notable success in continuous control tasks. However, it faces challenges that limit its applicability to a wider array of tasks, including sparse rewards and limited exploration. Recently, the integration of evolutionary algorithms (EAs) with deep reinforcement learning has emerged as a significant area of research. Evolutionary reinforcement learning (ERL) methods can help address specific challenges inherent in conventional reinforcement learning algorithms. However, the introduction of evolutionary computation algorithms increases the number of hyperparameters, and sensitivity to these hyperparameters continues to pose a significant challenge. This paper proposes an evolutionary reinforcement learning method incorporating evolutionary mutation rates. This method integrates a self-adaptive mutation rate mechanism into the ERL framework, which maintains two populations: one consisting of individuals (agents) and the other one comprising mutation rates. This represents our original contribution to this research. The actor population is categorized into several groups, each assigned a specific mutation rate. After mutation, the mutation rate of the population evolves based on the performance of the mutations within the actor population. This approach addresses the challenge of selecting mutation rates in ERL. Experimental results demonstrate superior performance compared to the standard ERL framework across six continuous control tasks.

The increasing penetration of renewable energy sources poses significant challenges for modern power systems, particularly in supply-demand balance and peak regulation. Load aggregators (LAs) play a crucial role by integrating small to medium-sized loads and coordinating demand response (DR). However, previous research works ignored the inherent coupling between price-maker LAs’ decision-making of bidding price and quantity in the ancillary service market and decision-making of incentive price in DR. This study introduces a joint bidding and incentivizing model for a price-maker LA participating in a peak-regulation ancillary service market (PRM) and developing an incentive-based demand response (IBDR), where the LA’s objective is to maximize its long-term cumulative payoff. In order to solve this complex joint decision-making optimization problem more effectively and efficiently, a model-free multi-task multi-agent deep reinforcement learning-based (MTMA-DRL-based) method incorporating a shared, centralized prioritized experience replay buffer (PERB) is proposed. Case studies in real-world settings confirm that the proposed model effectively captures the interdependence between bidding price, bidding quantity, and incentive price decisions. The proposed MTMA-DRL-based method is also proven to outperform existing methods.

Workflow Scheduling is a crucial challenge in cloud computing as different task dependencies involved in workflows which makes task scheduling process more complex and incurs huge energy consumption for Cloud Service Provider (CSP). Inefficient scheduling of workflows onto VMs without considering runtime capacity of tasks leads to increase in consumption of energy, makespan and operational costs for CSP. It is also important to consider the cooling costs for CSP as and when the temperature increases at the datacenter then there is a definite overhead on CSP to invest in cooling costs which in turn leads to increase of costs for both CSP and cloud users. Therefore, to mitigate these operational costs and to reduce energy consumption in cloud computing an energy-temperature aware workflow scheduler based on deep reinforcement learning technique DQN is proposed which calculates priorities of tasks based on their computational capacity and calculates priorities of datacenters based on temperature. After calculation of task, datacenter priorities these are given as input to scheduler induced with DQN model to schedule tasks according to the evaluated priorities. Proposed energy- temperature aware workflow scheduler based on Deep Reinforcement learning (ETAWSDRL) is implemented on workflowsim. Proposed ETAWSDRL is evaluated using scientific workflows(Epigenomics, LIGO, Cybershake, Montage). It is evaluated using existing approaches FCFS, PSO, ACO. Results of proposed ETAWSDRL shown significant improvement over compared approaches while minimizing makespan, utilization of resources, energy consumption, scheduling overhead by 86.7%, 78.32%, 87.1%, 36.2% respectively.

Due to limited network resources in internet of vehicles (IoV), vehicle’s heterogenous data, communication and computing resources significantly impact the training delay and model accuracy of federated learning (FL). To enhance resource utilization in IoV and mitigate the impact of limited resources on FL, we propose a resource allocation strategy to optimize computing and communication resource distribution within IoV. Additionally, A vehicle selection strategy based on vehicle data, communication and computing resources is introduced to reduce the impact of resource and data heterogeneity on FL. To further enhance FL training efficiency, a hierarchical FL algorithm that integrates vehicle selection and resource allocation based on deep deterministic policy gradients is designed. The vehicle selection and resource allocation problems are modeled as a Markov decision process. Deep reinforcement learning is adopted to find the optimal strategy. Simulation results demonstrate that the proposed algorithm achieves rapid convergence. It significantly reduces FL training latency while ensuring training accuracy.

As the threat landscape for Operational Technology (OT) and Supervisory Control and Data Acquisition (SCADA) systems grows more complex, there is a pressing need for intrusion detection systems that can dynamically adapt to evolving attack patterns. Traditional Machine Learning (ML) approaches often require frequent manual retraining and struggle to respond efficiently to these dynamic threats. Deep Reinforcement Learning (DRL) models present a promising solution, offering autonomous learning capabilities, adaptability to diverse scenarios with minimal human intervention, and enhanced intrusion detection for Industrial Control Systems (ICS). This paper presents a novel investigation into the application of various DRL models, including Deep Q-Network (DQN), Double Deep Q-Network (DDQN), Dueling Double Deep Q-Network (D3QN), REINFORCE, Advantage Actor-Critic (A2C), and Proximal Policy Optimization (PPO), for network intrusion detection in ICS. Performance comparisons with traditional ML methods are conducted using relevant metrics. To assess their effectiveness without a live environment, labeled pre-recorded intrusion datasets are utilized, with tailored adaptations for DRL model training outlined. These adaptations include generating data samples in mini-batches, integrating small discount factors, and employing straightforward reward functions. Comprehensive results underscore the efficacy of DRL models in bolstering the detection of advanced cyberattacks within OT environments, surpassing conventional ML approaches.

Recent advancements in deep neural network (DNN) technology are enhancing the utilities of machine learning and meeting potential demands for real-world applications. Deep reinforcement learning (DRL) is a key component in realizing complex decision-making and control by machine learning. However, training DRL-based policies is challenging due to the high costs and risks associated with interactions with real environments. Frameworks to avoid adversarial vulnerability and to achieve robustness are promising solutions to this problem, as they consider worst-case inputs during the evaluation and training phases. In the field of adversarial DRL, regularization methods are known to be effective in maintaining output consistency even when suffering from adversarial perturbations. However, these methods often introduce an undesired bias that conflicts with the original DRL objective, that is, to maximize accumulated rewards. In this study, we propose a new technique that mitigates the undesirable bias effect by orthogonalizing the regularization term with the DRL objective. This approach allows for the use of relatively large adversarial perturbations during training, resulting in more robust policies without significant computational overhead. We evaluate our technique on benchmarks for both discrete and continuous action domains, including Atari 2600 and PyBullet, by integrating it into popular DRL methods such as deep Q-network (DQN), deep deterministic policy gradient (DDPG), and soft actor-critic (SAC). The results demonstrate that our proposed technique is either superior to or competitive with baseline and state-of-the-art methods across all settings, particularly improving robustness by at least twice scores in DQN and DDPG under the adversarial perturbations.

This paper designs a novel Hierarchical Federated Learning (HFL) management scheme, enabled by deep reinforcement learning (DRL), for multi-access edge computing (MEC) environments to accelerate convergence. To do this, the proposed scheme controls the number of local updates performed by mobile devices (MDs) and the number of intermediate aggregations at the MEC server before data is transmitted to the cloud for global aggregation. This optimization aims to i) mitigate the straggler effect by balancing training times between MEC and cloud servers, and ii) reduce the risk of overfitting by avoiding excessive reliance on faster MDs. Additionally, to improve the efficiency of DRL, Bayesian optimization is employed to initialize action values, thereby avoiding inefficient exploration of actions. Extensive simulations demonstrate that our proposed scheme outperforms various benchmarks in terms of test accuracy and training time.

With the Internet of Things (IoT) rapidly increasing in popularity, it is necessary to progress Beyond fifth-generation (B5G) applications to support access to a large number of devices under the strict Ultra-Reliability and Low-Latency Communication (URLLC) requirements, which are intimately tied to strict delay requirements. This article focuses on minimizing co-channel interference, implementing efficient transmitter power, satisfying the Quality-of-Service (QoS), and assigning the energy harvesting requirement of radio frequencies using a Power Splitting Ratio (PSR) to maximize energy efficiency. This article proposes an adaptive approach algorithm for power allocation and PSR to reduce energy consumption caused by a nonconvex optimization problem, to achieve a high strict level of URLLC, QoS aware immediate reward, and huge connection. In addition, the proposed Deep Reinforcement Learning-Based Reduction Action Space (DRL-RAS) allows achieving a high degree of massive access to a huge number of devices by tackling the rate requirement of IoT devices with high intelligence to increase learning efficiency by maximizing the immediate reward function and ensuring strict URLLC. The simulation’s results demonstrate how DRL-RAS decreases co-channel interference based on decision selection to provide strict URLLC. The performance of energy efficiency and QoS are based on achieving a highly strict level of URLLC. The suggested DRL-RAS training strategy performs better, gaining QoS satisfaction levels of 17.21% and 8.67% when the needed QoS rises from 0.88 bits/sec/Hz to 1 bit/sec/Hz. Moreover, the proposed DRL-RAS eliminates error prediction during the training process by updating the weight of DRL-RAS to minimize the loss function and maximize energy efficiency.

Vehicular edge computing (VEC) has emerged as a solution that places computing resources at the edge of the network to address resource management, service continuity, and scalability issues in dynamic vehicular environments. However, VEC faces challenges such as task offloading, varying communication conditions, and data security. To tackle these challenges, federated learning (FL), a distributed machine learning framework that allows multiple clients to collaboratively train a global model without sharing their data, is utilized. However, vehicular clients have characteristics such as non-independent and identically distributed (non-IID) data, diverse communication capabilities, and high mobility, which pose difficulties for model convergence. A dynamic and optimal client selection method is required to address VEC and FL challenges. Therefore, in this paper, we propose a distributed client selection method with multi-objectives that can dynamically adapt to changing conditions. This method combines fuzzy logic with deep reinforcement learning (DRL) based deep Q-network (DQN). Initially, the fuzzy logic approach infers client candidates based on the stability of communication links. Subsequently, the DQN approach selects the final clients by considering the objectives of maximizing model accuracy and minimizing processing time and communication overhead. Unlike conventional methods, the proposed method provides an efficient solution that balances different objectives and improves model performance by ensuring comprehensive network coverage. Consequently, the proposed method achieves higher model accuracy, lower processing time, and reduced communication overhead compared to conventional methods.

Satellite mobile communication networks can provide communication services in areas beyond the coverage of terrestrial cellular networks. Ground relay stations for satellite networks is an effective solution to improve the coverage quality and to ensure connectivity between regular ground terminals and the satellite network. Without power grid coverage, the ground relay station can only be powered by green energy generation devices. So the problem of energy-saving under sleep strategy is worthy of studying.This paper proposes an energy-saving sleep control method for ground relay stations through Double Deep Q Network (DDQN) algorithm. Based on the traditional rule-based method that is only set for residual energy, it also considers the impact of weather factors on solar and wind power generation on the energy efficiency of relay stations. The research results show that by considering multiple influencing factors at the same time, the method used in this paper can effectively save non-communication energy consumption at low loads, ensure the satellite communication needs of most users, and improve the operational energy efficiency of the relay station over a period of time.

End-to-end autonomous driving systems aim to address the perception, decision-making, and control challenges in a self-contained way, facilitating easier adaptation to new scenarios and improved scalability. However, existing end-to-end approaches often lack safety constraints and are limited to simple driving tasks. In this study, we propose a novel method that integrates safety constraints into deep reinforcement learning for end-to-end autonomous driving, capable of handling complex urban scenarios. To achieve this, we introduce safety information into the probabilistic graphical model(PGM) and learn it in conjunction with the reinforcement learning process. An auxiliary safety critic evaluates the safety performance of the ego vehicle and assists in training the driving policy. Through the safety critic, the learned policy is able to provide better actions to control the ego vehicle during task completion. A sequential latent environment model is introduced and learned jointly with the reinforcement learning process. Compared to existing methods, our approach not only incorporates predefined safety constraints but also offers end-to-end training with enhanced interpretability through the latent model. This interpretability provides insights into decision-making and aids in debugging. We evaluate our algorithm using the CARLA simulator for autonomous vehicles, demonstrating its effectiveness in handling complex urban scenarios and providing enhanced safety information, even in crowded environments with multiple surrounding vehicles. The code (https://github.com/houchangmeng/safe-e2e-driving) will be made publicly available after the article is published.

Artificial intelligence (AI) in autonomous vehicles (AVs) is gaining much focus on safety, comfort, and efficiency. The goal is to provide the driver with assistance and mimic human driving. Deep reinforcement learning (DRL), specifically the deep deterministic policy gradient (DDPG) is often used for longitudinal velocity control, utilizing the velocities of the vehicles and the distance between them. DDPG has been found effective in many studies when it is embedded with a self-supervised learning (SSL) method. In this paper, a framework for longitudinal velocity control is proposed using SSL and DDPG frameworks. The inputs of the DDPG networks have been replaced by the outputs of the SSL network. The primary objective of this SSL is to enable the model to accurately predict future states based on current states and actions. The input features of the datasets have been modified so that the DDPG model can get additional information that helps the model make better predictions. These features include the distance between vehicles, ego vehicle velocity, acceleration, jerk, and lead vehicle relative velocity, and estimated velocity. Furthermore, a custom reward function is designed to account for safety, driving comfort, negative impact, driving aggressiveness, and fuel efficiency. In order to evaluate the model, the algorithm has been trained and tested on a variety of datasets, including simulated and real-world data. The analysis demonstrates that the new architecture maintains strong robustness across various datasets and outperforms the current state-of-the-art models.

Intelligent reflective surface (IRS) is speculated to be one of the key enabling technologies for the beyond fifth-generation (B5G) of wireless communication systems since it reconfigures the wireless propagation environment by tuning the incoming waveform’s phase shift and amplitude and effectively improve the efficiency and coverage of wireless networks. In this paper, we investigate the spectral efficiency (SE) of multiuser multiple-input single-output (MISO) system, where the direct channel between the user and the BS is blocked and the signal transmission between the user and the BS is assisted by IRS. To maximize the total SE of the system, we jointly optimize the phase shift matrix at the IRS and the user transmit power allocation under the constraints of the system’s total power. Since the optimization problem is nonconvex and the objective function cannot be obtained in closed-form, we develop the twin delayed deep deterministic (TD3) policy gradient algorithm by leveraging recent advances in deep reinforcement learning (DRL). The algorithm obtains the optimal solution by continuously interacting with the system environment and learning, in which a set of channel state information (CSI) vectors is generated in an offline way, and then these data sets are used to train the neural networks. The numerical results show that the proposed algorithm can greatly improve the system SE after training and learning. In addition, the system performance and network convergence speed can be effectively improved by adjusting the network parameters.

Mobile edge computing (MEC) represents a promising computing paradigm within Artificial Intelligence Generated Content (AIGC), offering users instant, customized, and personalized services. However, with the continued growth of the AIGC user base and the expansion of service demands, edge computing nodes are required to handle an increasing number and complexity of offloading tasks, leading to significant energy consumption issues in edge systems. This paper introduces a distributed task offloading framework (EE-A2C) that utilizes the Advantage Actor-Critic algorithm to enhance energy efficiency in edge cloud environments. This framework facilitates distributed interactions among multiple agents and edge environments with communication queues, aiming to minimize average energy consumption and latency for AIGC users. Secondly, to achieve adaptive and efficient task offloading decisions, we have developed a complex reward-sharing model based on latency and energy consumption. Finally, we have also incorporated LSTM to enhance the model’s ability to capture critical information about energy consumption, thereby promoting more robust decision-making. Compared to eight state-of-the-art energy-saving algorithms, the proposed EE-A2C framework more effectively utilizes the computing power of edge nodes, significantly reduces average energy consumption and latency, and enhances the energy efficiency of the edge cloud system.

To effectively address the heterogeneous service requirements of diverse users, the Fifth Generation (5G) has introduced network slicing technology, which partitions physical resources into multiple virtual end-to-end network slices, allowing for the dynamic allocation of these slices. However, integrating network slicing with the 5G New Radio (NR) frame structure and allocating resources to meet Quality of Service (QoS) requirements, while ensuring user fairness, remains a challenging task. To tackle this challenge, a double layer resource configuration framework is proposed based on the 5G NR standard. Subsequently, an optimization problem is formulated to maximize user satisfaction taking into account their distinctive QoS requirements. In the upper layer of the framework, we leverage the Dueling Double Deep Q Network (D3QN) algorithm to optimize resource allocation from the base station to the network slices. In the lower layer, we employ the parameter tuning method to optimize the resource allocation of the users within each slice. These two steps are combined to form the Double Layer Distribution-D3QN (DLD-D3QN) algorithm. The effectiveness and superiority of the proposed algorithm is demonstrated through extensive simulations.

Layered water injection is an important means of oilfield development. However, the process is fraught with challenges due to the complexity of the subsurface environment and the variability in water absorption properties across different strata. The water injection flow rate of each layer is affected by various factors, resulting in a typical incomplete system and extremely complex measurement and adjustment of layered water injection. The reasons include development plans, water absorption properties of each layer, and differences in the structure of underground water injection devices. The combined effect of these factors leads to the typical incomplete system and complexity of the measurement and regulation of the layer section of layered water injection. In this paper, a reinforcement learning-based stratified water injection control algorithm was proposed to solve the problem of stratum flow scheduling in the complex environment of wellbore during stratified water injection. The reinforcement learning algorithm is combined with the differential pressure stratified water injection control algorithm, and a reasonable reward function is set according to the injection error to improve the algorithm control accuracy and efficiency. In order to evaluate the performance of the algorithm, a gym-based simulation environment is established to simulate the nonlinear stratigraphic environment under stochastic conditions, so that the model has a better generalization performance. Compared with other algorithms, the proposed stratified water injection control algorithm saves 41.94% of training time, the average injection error is less than 5% in the water injection environment with different number of stratum segments, the average success rate is more than 90%, and there is 85% probability to reach the injection target within 1-5 steps, which provides a more excellent performance in terms of control accuracy and adjustment speed. The algorithm has an important guiding role in the flow scheduling control of injection wells and realizing the automation of layered water injection process. Our code will be available online at: https://github.com/HJZ-hub/ASWI.

Object detection is an essential step in various applications. After deep learning appeared, convolutional neural networks or transformers have shown significant improvement in object detection compared to statistically motivated algorithms. But, they still require improvement in multiple aspects. One is to maintain detection performance in unseen environments without retraining using images and labels from unseen environments. The other is to reduce the tight requirements of labels. In object detection, a bounding box is usually used as a label for an object. In this paper, we propose an object detection algorithm that requires only images and the number of objects on images as labels. We approach the problem with deep reinforcement learning. The proposed algorithm uses an actor-critic algorithm that can produce continuous action. We make an actor model to produce multiple bounding boxes, and a critic model evaluates well as training goes on. Also, we propose a reward model using a pre-trained model trained with an object detection dataset. The proposed algorithm requires only images and the number of objects on images, not bounding boxes. We show that the proposed algorithm gives a comparable result to the transformer-based approach through experiments. Also, it can adapt to unseen environments by only using images and the number of objects on images.

This work presents a novel control paradigm to improve the Direct Current Regulation (DCR) of two-level inverters that are connected to the grid with LCL filters. The Deep Reinforcement Learning (DRL) based Deep Deterministic Policy Gradient (DDPG) algorithm is utilized to address the constraints of traditional control methods, such as Proportional Integral (PI) controllers and Model Predictive Control (MPC). The suggested method tackles challenges like as non-linearity, model dependency, and parameter fluctuations, which have a substantial impact on the performance of the DCR of grid connected two-level inverters of electric power system. The DDPG algorithm offers a flexible control technique that enables adaptive learning and optimization of policies. The results are validated in Real time mode Hardware In Loop (HIL) using Opal-RT & Texas instrument launchpad. Simulations performed on the MATLAB platform provide a reliable testing environment to assess the effectiveness of the proposed controller compared to traditional alternatives. The simulation results clearly show that the DDPG-based controller performs better than any other controller. This strategy surpasses traditional methods, demonstrating an increased resistance to reliance on specific models and uncertainties in parameters.

Facial super-resolution (FSR) is a critical research area whose goal is to improve visual quality by converting low-resolution facial images to high resolution ones. Research in FSR has come a long way thanks to advances in deep learning technologies. However, there is still a need to develop effective methods for revealing facial details and preserving the overall appearance. For this purpose, a new approach called Deep Reinforcement Learning Based Super Resolution of Face Regions (DRL-SRFR) is proposed. It is based on Deep Reinforcement Learning (DRL) and Deep Residual Dense Block (RRDB) architectures. In the DRL part of the method, new regions that need attention are identified at each step using the repeated visual attention methodology. The details in different parts of the face image are iteratively improved to produce more natural and high-quality face images. In addition, with the stochastic action-taking process, the decision-making process is made flexible by focusing on important facial regions. The focused region is improved with the RRDB structure using dense connections and residual learning. Experiments and ablation studies show that the developed model provides a significant advantage over existing methods in improving local details and preserving appearance integrity.

As Unmanned Combat Aerial Vehicle (UCAV) continue to play an increasingly pivotal role in modern aerial warfare, enhancing their intelligence levels is imperative for global military advancement. Despite notable progress in employing deep reinforcement learning for autonomous air combat maneuver decision-making, existing methods grapple with subpar performance, sluggish training, and susceptibility to local optima. Therefore, this paper proposes a new air combat maneuver decision algorithm based on Proximal Policy Optimization (PPO). Firstly, we establish a UCAV adversarial model and design a dual observation space. Secondly, we develop an Actor-Critic network based on Bidirectional Long Short-Term Memory (BiLSTM) and Multi-Head Self-Attention (MHSA), which better handles high-dimensional information with temporal correlations in air combat situations. Thirdly, we propose an action selection method based on Parallel Monte Carlo Tree Search with Watch the Unobserved (WU-PMCTS) to assist the algorithm in making more effective maneuver decisions. Fourthly, we design a Dynamic Reward Evaluation (DRE) method to dynamically adjust the weights of various rewards according to different adversarial situations, improving algorithm performance. Finally, we introduce an Advantage Prioritized Experience Replay (APER) to sample according to the sample advantage values, enhancing algorithm training efficiency. Experimental results from ablation and comparative experiments demonstrate the superiority of the proposed algorithm over PPO and other mainstream algorithms, with a 0.32 increase in average return and a 36% increase in win rate.

This study addresses the limitations of traditional active and semi-active suspension control systems in terms of adaptability and nonlinear handling, by exploring the potential of Deep Reinforcement Learning (DRL) techniques. Initially, a framework based on the Twin Delayed Deep Deterministic policy gradient (TD3) specific to active suspension systems was developed. Building on this, an enhanced TD3 algorithm, TD3-PSC (Physically Safe Constraint TD3), incorporating physical safety constraints was proposed. The TD3-PSC algorithm extends the state space to enhance understanding of suspension dynamics and improve adaptability. To accommodate the physical constraints and actuator characteristics inherent in suspension systems, TD3-PSC introduces guided training with real physical constraints and employs immediate termination and high penalty mechanisms to ensure safety and practicality of the algorithm. The simulation results demonstrate that TD3-PSC significantly outperforms the linear quadratic regulator (LQR), deep deterministic policy gradient (DDPG), and standard TD3 baseline, achieving improvements in control performance of 73.81%, 43.72%, and 32.14% under standard Class C road conditions, respectively. Additionally, it exhibits excellent generalization capabilities.

Pan-Tilt Platform is essential for visually tracking moving targets over a wide range of regions. Due to the target’s unknown motion state and the tracking environment’s complexity, controlling the pan-tilt platform to keep the target at the center of the video frame becomes a challenging task. Therefore, a motion control technique relies on a model-based reinforcement learning algorithm is proposed. The pinhole model of the camera is used to convert the pixel coordinate to rotation angle. In addition to calculate the commanded angular speed of the pan-tilt platform using a car-following control model, the performance of the proposed models is performed by adopting a Deep Q-learning algorithm. The proposed motion control model is evaluated considering random movements of the target, and the experiment results prove the control performance in terms of tracking accuracy and robustness.

We consider the problem of optimal unsignalized intersection management, wherein we seek to obtain safe and optimal trajectories, for a set of robots that arrive randomly and continually. This problem involves repeatedly solving a mixed integer program (with robot acceleration trajectories as decision variables) with different parameters, for which the computation time using a naive optimization algorithm scales exponentially with the number of robots and lanes. Hence, such an approach is not suitable for real-time implementation. In this paper, we propose a solution framework that combines learning and sequential optimization. In particular, we propose an algorithm for learning a shared policy that given the traffic state information, determines the crossing order of the robots. Then, we optimize the trajectories of the robots sequentially according to that crossing order. This approach inherently guarantees safety at all times. We validate the performance of this approach using extensive simulations and compare our approach against 5 different heuristics from the literature in 9 different simulation settings. Our approach, on average, significantly outperforms the heuristics from the literature in various metrics like objective function, weighted average of crossing times and computation time. For example, in some scenarios, we have observed that our approach offers up to 150% improvement in objective value over the first come first serve heuristic. Even on untrained scenarios, our approach shows a consistent improvement (in objective value) of more than 30% over all heuristics under consideration. We also show through simulations that the computation time for our approach scales linearly with the number of robots (assuming all other factors are constant). We further implement the learnt policies on physical robots with a few modifications to the solution framework to address real-world challenges and establish its real-time implementability.

Industrial robots play a crucial role in a wide range of industrial processes. Because of the complexity of the work environment in which these systems are deployed, more robust and accurate control methods are required. Deep reinforcement learning (DRL) is a comprehensive approach that does not require an initial source of structured data for its learning process. Instead, DRL generates its own data based on its experiences within a work environment. To generate its own data, DRL requires integration with virtualized environments provided by simulators. These tools must include scenarios in industrial contexts and allow integration with machine learning tools, among other capabilities. Currently, several platforms support the simulation of various scenarios and generation of synthetic data, thus facilitating the development of end-to-end systems based on artificial intelligence, such as DRL. This article presents an extensive review of the software tools applied to DRL-based control systems for robotic manipulators. The selection of these tools is based on their efficiency, scalability, and compatibility with contemporary industrial standards and offers insights into their practical application in real-world scenarios. This study established a complete framework for designing and developing control systems for robotic manipulators using end-to-end DRL. This framework outlines the tools in detail, including simulators, APIs, libraries, and methods, and their interactions with each other. Additionally, it discusses the practical implications of this framework, highlighting its potential applications in industry and addressing some of the challenges and limitations encountered in applying DRL to complex robotic systems.

Reinforcement learning (RL) has seen an uptick in research interest in recent years, with many papers published in a plethora of different fields, topics and applications. A lot of that can be attributed to the recent advancements in machine learning (ML) and deep learning (DL) as a whole, the power of deep neural networks and the incorporation of them into reinforcement learning algorithms and techniques. This marriage of deep neural networks and traditional reinforcement learning methods has enabled reinforcement learning algorithms to achieve incredible results in use-cases that were previously deemed to be too complex. One of these highly complex scenarios is the realm of flight. In the last few years the number of papers published in this field has increased dramatically, but there is a fairly big divide between two major categories (quadcopters vs. fixed-wing aircraft). There is published work focusing on reinforcement learning using quadcopters, also known as drones, and there is published work focusing on fixed-wing aircraft, the more traditional aircraft, commonly referred to as airplanes. This paper focuses on the latter and will give an overview over the current state of reinforcement learning in the field of fixed-wing aviation. It will look into available toolkits usable for research as well as give an overview of different fields and tasks within reinforcement learning for fixed-wing aircraft control tasks. It contributes by providing detailed information about the methodologies used in each paper, while pointing out each paper’s contributions as well. A total of 48 papers on fixed-wing RL and airplane control are discussed. With the main focus lying on papers published post 2015, after the publication of Mnih et al.’s impactful deep RL paper.

Multi-Agent Reinforcement Learning (MARL) is extensively utilized for addressing intricate tasks that involve cooperation and competition among agents in Multi-Agent Systems (MAS). However, learning such tasks from scratch is challenging and often unfeasible, especially for MASs with a large number of agents. Hence, leveraging knowledge from prior experiences can effectively expedite the MARL learning process. Prior work has shown that we successfully facilitated transfer learning for MARL by consolidating various state spaces into fixed-size inputs, enabling a single unified deep-learning policy applicable to several scenarios within the StarCraft Multi-Agent Challenge (SMAC) environment. In this study, we expand SMAC to Multi-Player enabled SMAC (MP-SMAC) by enabling the dynamic selection of training opponents and introducing a co-evolving MARL framework, which creates a co-evolutionary arena where multiple policies learn simultaneously. Our arena comprised the simultaneous training of multiple policies in diverse scenarios, pitting them against both static AI opponents and their peers within MP-SMAC. Furthermore, we integrate co-evolution with curriculum transfer learning into Co-MACTRL framework, enabling our MARL policies to systematically acquire knowledge and skills across predetermined scenarios organized by varying difficulty levels, including evolving opponents. The results revealed significant enhancements in MARL learning performance, demonstrating the advantage of leveraging the co-evolving opponents and maneuvering skills obtained from different scenarios. Additionally, the Co-MACTRL learners consistently attained high performance across a range of SMAC scenarios, showcasing the robustness and generalizability of Co-MACTRL.

The power grids nowadays are facing increasing complexity and uncertainty due to the continuously growing penetration of renewable energy sources, such as photovoltaic (PV) and wind power, as well as the emerging uncertain and dynamic nature of demand-side factors such as electric vehicles, portable energy storage, etc. To effectively manage these challenges, artificial intelligence (AI) technologies, particularly model-free deep reinforcement learning (DRL), have risen as a powerful tool for power system control. This paper presents an in-depth review of the state-of-the-art applications of model-free DRL in power system control. The review is focused on various model-free DRL approaches utilized in addressing uncertainty caused by stochastic factors from renewable generations, demand-side dynamics, and power system contingencies. Specifically, it investigates how model-free DRL techniques are employed to facilitate decision-making in the frequency control, voltage control, and optimal power flow control in the grid. The benefits, challenges, and limitations of these technologies are revealed, shedding light on recent advancements in the field and showcasing the performance of these methods across diverse power system scenarios. By synthesizing the findings from extensive research, this paper also highlights the limitations, key future research directions, and recommendations for deploying model-free DRL in power system control.

Network virtualization is a demanding feature in the evolution of future Internet architectures. It enables on-demand virtualized resource provision for heterogeneous Virtual Network Requests (VNRs) from diverse end users over the underlying substrate network. However, network virtualization provides various benefits such as service separation, improved Quality of Service, security, and more prominent resource usage. It also introduces significant research challenges. One of the major such issues is allocating substrate network resources to VNR components such as virtual machines and virtual links, also named as the virtual network embedding, and it is proven to be $\mathbb {N}\mathbb {P}$ -hard. To address the virtual network embedding problem, most of the existing works are 1) Single-objective, 2) They failed to address dynamic and time-varying network states 3) They neglected network-specific features. All these limitations hinder the performance of existing approaches. This work introduces an embedding framework called Intelligent Deep Reinforcement Learning (DRL) Strategy for effective virtual network embedding of an online VNRs (InDS). The proposed InDS uses an actor-critic model based on DRL architecture and Graph Convolutional Networks (GCNs). The GCN effectively captures dependencies between the VNRs and substrate network environment nodes by extracting both network and system-specific features. In DRL, the asynchronous advantage actor-critic agents can learn policies from these features during the training to decide which virtual machines to embed on which servers over time. The actor-critic helps in efficiently learning optimal policies in complex environments. The suggested reward function considers multiple objectives and guides the learning process effectively. Evaluation of simulation results shows the effectiveness of InDS in achieving optimal resource allocation and addressing diverse objectives, including minimizing congestion, maximizing acceptance, and revenue-to-cost ratios. The performance of InDS exhibits superiority in achieving 28% of the acceptance ratio and 45% of the revenue-to-cost ratio by effectively managing the network congestion compared to other existing baseline works.

In a MicroGrid (MG) equipped with a Battery Energy Storage System (BESS), an Energy Management System (EMS) plays a crucial role in predictive controlling BESS operations for optimal power flow among uncertainties from renewable energy resources and heavy loads, such as solar photovoltaic systems and electric vehicles, respectively. State-of-the-art EMS designs have integrated Deep Reinforcement Learning (DRL) for EMS development and Probabilistic Power Flow (PPF) for preventing the violation of power system constraints while accounting for all uncertainties. However, using PPF to handle uncertainties alongside training a single-agent DRL provides the optimal solution for addressing all uncertain scenarios, but not the best solution for each scenario. Moreover, employing a single-agent DRL yields a low performance in predictive controlling of BESS operations. To address these challenges, a multi-agent DRL based on Deep Deterministic Policy Gradient (DDPG) is proposed. This method divides the roles of each agent for predicting 24-hour-ahead actions in BESS control based on changing 24-hour-ahead MG behavior every hour. Furthermore, MG parameters are randomly sampled to retain MG uncertainties instead of relying on PPF for uncertainty mitigation. Consequently, multi-agent DDPG with random sampling can directly learn from the MG environment and provide the best solution for each scenario. Simulation results demonstrate that the proposed method can reduce training computational time by 92.84%, provide a higher value of the summed mean of 24-hours-ahead reward by 1.50% to 28.37%, and achieve a lower mean daily total related cost by 9.22% compared to applying the state-of-the-art method.

An intelligent reflecting surface (IRS) is a promising technology for future wireless communication. It comprises many hardware-efficient passive elements. The applications of unmanned aerial vehicles (UAVs) have expanded beyond military missions owing to their mobility, maneuverability, flexibility, ease of deployment, and cost-effectiveness. Combining IRS with UAVs, known as UAV-mounted IRSs (IRS-UAV), has gained significant attention owing to the unique advantages offered by both technologies. Wireless communication systems face critical challenges in physical layer security, particularly cell-free massive multiple-input multiple-output (MIMO) systems (CFMM). This study investigated physical layer security (PLS) in an IRS-UAV-assisted CFMM, involving multiple IRS-UAVs, access points, users, and passive eavesdroppers. To maximize the average secrecy downlink rate, this study proposes an optimization algorithm using deep reinforcement learning based on a deep deterministic policy gradient (DDPG) that achieves at least one locally optimal solution. However, this approach results in a relatively high computational complexity. A second Approach Is introduced to address this: an alternating optimization algorithm combining inner approximation (IA) methods and an advanced DDPG algorithm with a warm-up technique. The simulation results demonstrated the efficiency of both approaches in resolving complex optimization problems. Furthermore, the numerical findings confirmed that the proposed alternating optimization algorithm exhibited competitive performance and significantly reduced computational complexity compared with the DDPG-based approach.

Industry 4.0 incorporates the integration of cloud computing, Industrial Internet of Things (IIoT), and modern communication technologies within the industrial automation systems. Various devices with different network requirements of high reliability and low latency, rely on connectivity. The 5G and Beyond (B5G) software-defined architecture facilitates Network Function Virtualization (NFV), which is an essential solution for fulfilling these stringent demands. NFV allows for the implementation and control of Virtual Network Functions (VNFs) in dynamic network environments. VNF placement optimization has been extensively studied in the 5G perspective outside the industry environment with a focus on minimizing delay and cost, increasing VNF reliability, and increasing resource efficiency. However, the complex dynamics of the wireless channel in industrial environments have a considerable impact on the essential delay factors that are important for optimizing the deployment of VNFs. This study focuses on modeling a Wireless Sensor Network (WSN) based Industry 4.0 factory automation scenario at mmWave band, formulating an optimization problem to minimize overall delay while considering packet loss rate in the 5G industrial wireless channel. The optimization problem is formulated as a Markov Decision Process (MDP) and two Reinforcement Learning (RL) based algorithms AVP-Q and AVP-DQN are proposed for optimizing the VNF placement. The proposed algorithms are extensively evaluated against the Value Iteration algorithm which assumes a completely known MDP model and two other algorithms from the literature. The simulated results show that AVP-DQN outperforms existing algorithms for this scenario by 39% and 22.6% and the achieved performance is only close to that of the Value Iteration algorithm.

As artificial intelligence (AI) technology advances rapidly, its increasing involvement in military defense fosters intelligent air combat domain development. The Intelligent Flight Controller (IFC) is a crucial technology and foundation for intelligent air combat decision-making systems. Controlling 6 Degree-of-freedom (DOF) aircraft in close-to-real-world environments requires an adaptable and dynamic decision-making controller. Most IFC researches focus on simplistic flight trajectory design and validation, while air combat requires aircraft that can perform complex tactical maneuvers. Deep reinforcement learning (DRL) provides a suitable technical paradigm. However, DRL suffers from sparse rewards, insufficient supervisory signals, low sampling efficiency, and slow convergence. In contrast, Large Language Model (LLM) possesses abundant knowledge about the real world, contextual understanding, and reasoning capabilities. By leveraging this, LLM can serve as prior knowledge for DRL, thereby reducing DRL training time. This paper proposes an LLM-guided deep reinforcement learning framework for IFC, which utilizes LLM-guided deep reinforcement learning to achieve intelligent flight control under limited computational resources. LLM provides direct guidance during training based on local knowledge, which improves the quality of data generated in agent-environment interaction within DRL, expedites training, and offers timely feedback to agents, thereby partially mitigating sparse reward issues. Additionally, we present an effective reward function to comprehensively balance the aircraft coupling control to ensure stable, flexible control. Finally, simulations and experiments show that the proposed techniques have good performance, robustness, and adaptability across various flight tasks, laying a foundation for future research in the intelligent air combat decision-making domain.

Deep reinforcement learning (DRL) can learn an agent’s optimal behavior from the experience it gains through interacting with its environment. However, since the decision-making process of DRL agents is a black-box, it is difficult for users to understand the reasons for the agents’ actions. To date, conventional visual explanation methods for DRL agents have focused only on the policy and not on the state value. In this work, we propose a DRL method called Mask-Attention A3C (Mask A3C) to analyze agents’ decision-making by focusing on both the policy and value branches, which have different outputs. Inspired by the Actor-Critic method, our method introduces an Attention mechanism that applies mask processing to the feature map of the policy and value branches using mask-attention, which is a heat-map representation of the basis for judging the policy and state values. We also propose the introduction of a Mask-attention Loss to obtain highly interpretable mask-attention. By introducing this loss function, the agent learns not to gaze at regions that do not affect its decision-making. Our evaluations with Atari 2600 as a video game strategy task and robot manipulation as a robot control task showed that visualizing the mask-attention of an agent during its action selection facilitates the analysis of the agent’s decision-making. We also investigated the effect of Mask-attention Loss and confirmed that it is useful for analyzing agents’ decision-making. In addition, we showed that these mask-attentions are highly interpretable to the user by conducting a user survey on the prediction of the agent’s behavior.