Search

To help Latin dancers feel the rhythm of the sports dance, it is necessary to standardize the posture action during basic training. Therefore, it is important to study the method of correcting Latin dance posture action. The traditional Latin dance posture correction methods have some problems, such as the included angle error of head motion, the angle error of spine transformation, the error of fit between foot and ground and so on. In this paper, a Latin dance posture correction method using improved deep reinforcement learning in the Internet of things (IoT) is proposed. First, the Latin dance posture image acquisition architecture is constructed using IoT and binocular stereo vision to acquire Latin dance posture images and extract Latin dance posture features. Second, the channel attention module in the deep learning network is improved, and the Latin dance posture diagnosis model is constructed based on the action feature extraction results using the improved deep robust chemical network. Finally, the action correction coefficients are calculated according to the Latin dance posture diagnosis results to realize the Latin dance posture correction. The results showed that after the application of the proposed correction method, including angle error of head movement, the spine transformation angle error and fit between foot and ground error of the participants’ motions were kept below 1∘, and the frame position offset was 1.3cm. It indicates that the proposed method can effectively improve the degree of Latin dance posture specification. [ABSTRACT FROM AUTHOR]

This study delves into the critical need for generating real-world compatible data to support the application of deep reinforcement learning (DRL) in vehicle routing. Despite the advancements in DRL algorithms, their practical implementation in vehicle routing is hindered by the scarcity of appropriate real-world datasets. Existing methodologies often rely on simplistic distance metrics, failing to accurately capture the complexities inherent in real-world routing scenarios. To address this challenge, we present a novel approach for generating real-world compatible data tailored explicitly for DRL-based vehicle routing models. Our methodology centers on the development of a spatial data extraction and curation tool adept at extracting geocoded locations from diverse urban environments, encompassing both planned and unplanned areas. Leveraging advanced techniques, the tool refines location data, accounting for unique characteristics of urban environments. Furthermore, it integrates specialized distance metrics and location demands to construct vehicle routing graphs that represent real-world conditions. Through comprehensive experimentation on varied real-world testbeds, our approach showcases its efficacy in producing datasets closely aligned with the requirements of DRL-based vehicle routing models. It's worth mentioning that this dataset is structured as a graph containing location, distance, and demand information, with each graph stored independently to facilitate efficient access and manipulation. The findings underscore the adaptability and reliability of our methodology in tackling the intricacies of real-world routing challenges. This research marks a significant stride towards enabling the practical application of DRL techniques in addressing real-world vehicle routing problems. [ABSTRACT FROM AUTHOR]

One of the focal points in the field of intelligent transportation is the intelligent control of traffic signals (TS), aimed at enhancing the efficiency of urban road networks through specific algorithms. Deep Reinforcement Learning (DRL) algorithms have become mainstream, yet they suffer from inefficient training sample selection, leading to slow convergence. Additionally, enhancing model robustness is crucial for adapting to diverse traffic conditions. Hence, this paper proposes an enhanced method for traffic signal control (TSC) based on DRL. This approach utilizes dueling network and double q-learning to alleviate the overestimation issue of DRL. Additionally, it introduces a priority sampling mechanism to enhance the utilization efficiency of samples in memory. Moreover, noise parameters are integrated into the neural network model during training to bolster its robustness. By representing high-dimensional real-time traffic information as matrices, and employing a phase-cycled action space to guide the decision-making of intelligent agents. Additionally, utilizing a reward function that closely mirrors real-world scenarios to guide model training. Experimental results demonstrate faster convergence and optimal performance in metrics such as queue length and waiting time. Testing experiments further validate the method's robustness across different traffic flow scenarios. [ABSTRACT FROM AUTHOR]

AbstractAs greenhouse gas (GHG) emissions from road networks significantly impact the urban environment, there has been a growing interest in integrating environmental factors into traffic controls. Traffic signal control (TSC) is an effective method to reduce air pollution by managing traffic congestion. Accordingly, several recent studies have applied artificial intelligence to TSC to improve traffic efficiency. Despite these efforts, there are a few limitations on the existing studies. First of all, a number of recent studies focus only on traffic performance rather than environmental effects. However, in some cases, the objective function related to traffic efficiency can have an adverse effect on the environment. Furthermore, the variable nature of emissions based on traffic demand is rarely considered. To address these issues, we propose a multi-agent deep reinforcement learning-based TSC to minimize GHGs in traffic networks. To this end, we first identify vehicle’s stop and delay as key factors in traffic efficiency that critically influence air quality. Subsequently, we integrate these factors into our model’s reward function. To validate the proposed model, we conduct experiments under various demand scenarios featuring different traffic volume levels. The results show that the proposed model outperforms comparative control methods in two aspects: traffic performance and environmental sustainability. Especially at a high congestion level, the proposed model reduces average delay time by 11.9–16.1% and increases total throughput by 11.5%. Additionally, it significantly lowers total CO2 emissions by up to 8.9% and average emissions by up to 18.3%. [ABSTRACT FROM AUTHOR]

Highly dynamic Internet of Vehicles spectrum sharing can share spectrum owned by vehicle-to-infrastructure links through multiple workshop links to achieve efficient resource allocation. Aiming at the problem that the rapid variations in channel states in highly dynamic vehicular environments can make it challenging for base stations to gather and manage information about instantaneous channel states, we present a multi-agent deep reinforcement learning-based V2X spectrum access algorithm. The algorithm is designed to optimize the throughput of V2I user under V2V user delay and reliability constraints, and uses the experience gained from interacting with the communication environment to update the Q network to improve spectrum and power allocation strategies. Implicit collaborative agents are trained through an improved DQN model combined with dueling network architecture and long short-term memory network layers and public rewards. With lagged Q-learning and concurrent experience replay trajectories, the training process was stabilized and the non-stationarity problem caused by concurrent learning of multiple agents was resolved. Simulation results demonstrate that our presented algorithm achieves a mean successful payload delivery rate of 95.89%, which is 16.48% greater than that of the randomized baseline algorithm. Our algorithm obtains approximately the optimal value and shows performance close to the centralized brute force algorithm, which provides a better strategy for further minimizing the signaling overhead of the Internet of Vehicles communication system. [ABSTRACT FROM AUTHOR]

Although communication plays a pivotal role in achieving coordinated activities in multi-agent systems, conventional approaches often involve complicated high-dimensional messages generated by deep networks. These messages are typically indecipherable to humans, are relatively costly to transmit, and require intricate encoding and decoding networks. This can pose a design limitation for the agents such as autonomous (mobile) robots. This lack of interpretability can lead to systemic issues with security and reliability. In this study, inspired by common human communication about likely actions in collaborative endeavors, we propose a novel approach in which each agent's action probabilities are transmitted to other agents as messages, drawing inspiration from the common human practice of sharing likely actions in collaborative endeavors. Our proposed framework is referred to as communication based on action probabilities (CAP), and focuses on generating straightforward, low-dimensional, interpretable messages to support multiple agents in coordinating their activities to achieve specified cooperative goals. CAP streamlines our comprehension of the agents' learned coordinated and cooperative behaviors and eliminates the need to use additional network models to generate messages. CAP's network architecture is simpler than that of state-of-the-art methods, and our experimental results show that it nonetheless performed comparably, converged faster, and exhibited a lower volume of communication with better interpretability. [ABSTRACT FROM AUTHOR]

Global positioning systems often fall short in dense forest environments, leading to increasing demand for innovative localization methods. Notably, existing methods suffer from the following limitations: (1) traditional localization frameworks necessitate several fixed anchors to estimate the locations of targets, which is difficult to satisfy in complex and uncertain forestry environments; (2) the uncertain environment severely decreases the quality of signal measurements and thus the localization accuracy. To cope with these limitations, this paper proposes a new method of trajectory localization for forestry environments with the assistance of UAVs. Based on the multi-agent DRL technique, the topology of UAVs is optimized in real-time to cater for high-accuracy target localization. Then, with the aid of RSS measurements from UAVs to the target, the least squares algorithm is used to estimate the location, which is more flexible and reliable than existing localization systems. Furthermore, a shared replay memory is incorporated into the proposed multi-agent DRL system, which can effectively enhance learning performance and efficiency. Simulation results show that the proposed method can obtain a flexible and high-accuracy localization system with the aid of UAVs, which exhibits better robustness against high-dimensional heterogeneous data and is suitable for forestry environments. [ABSTRACT FROM AUTHOR]

The three-dimensional (3D) path planning problem of an Unmanned Aerial Vehicle (UAV) considering the effect of environmental wind in a dense city is investigated in this paper. The mission of the UAV is to fly from its initial position to its destination while ensuring safe flight. The dense obstacle avoidance and the energy consumption in 3D space need to be considered during the mission, which are often ignored in common studies. To solve these problems, an improved Deep Reinforcement Learning (DRL) path planning algorithm based on Double Deep Q-Network (DDQN) is proposed in this paper. Among the algorithms, the random obstacle training method is first proposed to make the algorithm consider various flight scenarios more globally and comprehensively and improve the algorithm's robustness and adaptability. Then, the linear soft update strategy is employed to realize the smooth neural network parameter update, which enhances the stability and convergence of the training. In addition, the wind disturbances are integrated into the energy consumption model and reward function, which can effectively describe the wind disturbances during the UAV mission to achieve the minimum drag flight. To prevent the neural network from interfering with training failures, the meritocracy mechanism is proposed to enhance the algorithm's stability. The effectiveness and applicability of the proposed method are verified through simulation analysis and comparative studies. The UAV based on this algorithm has good autonomy and adaptability, which provides a new way to solve the UAV path planning problem in dense urban scenes. [ABSTRACT FROM AUTHOR]

Algorithmic trading is playing an increasingly important role in the financial market, achieving more efficient trading strategies by replacing human decision-making. Among numerous trading algorithms, deep reinforcement learning is gradually replacing traditional high-frequency trading strategies and has become a mainstream research direction in the field of algorithmic trading. This paper introduces a novel approach that leverages reinforcement learning with human feedback (RLHF) within the double DQN algorithm. Traditional reward functions in algorithmic trading heavily rely on expert knowledge, posing challenges in their design and implementation. To tackle this, the reward-driven double DQN (R-DDQN) algorithm is proposed, integrating human feedback via a reward function network trained on expert demonstrations. Additionally, a classification-based training method is employed for optimizing the reward function network. The experiments, conducted on datasets including HSI, IXIC, SP500, GOOGL, MSFT, and INTC, show that the proposed method outperforms all baselines across six datasets and achieves a maximum cumulative return of 1502% within 24 months. [ABSTRACT FROM AUTHOR]

In order to enhance the trajectory tracking accuracy of distributed-driven intelligent vehicles, this paper formulates the tasks of torque output control for longitudinal dynamics and steering angle output control for lateral dynamics as Markov decision processes. To dissect the requirements of action output continuity for longitudinal and lateral control, this paper adopts the deep deterministic policy gradient algorithm (DDPG) for longitudinal velocity control and the deep Q-network algorithm (DQN) for lateral motion control. Multi-agent reinforcement learning methods are applied to the task of trajectory tracking in distributed-driven vehicle autonomous driving. By contrasting with two classical trajectory tracking control methods, the proposed approach in this paper is validated to exhibit superior trajectory tracking performance, ensuring that both longitudinal velocity deviation and lateral position deviation of the vehicle remain at lower levels. Compared with classical control methods, the maximum lateral position deviation is improved by up to 90.5% and the maximum longitudinal velocity deviation is improved by up to 97%. Furthermore, it demonstrates excellent generalization and high computational efficiency, and the running time can be reduced by up to 93.7%. [ABSTRACT FROM AUTHOR]

The virtual synchronous generator (VSG) is an important concept and primary control method in modern power systems. The penetration of power-electronics-based distributed generators in the power grid provides uncertainty and reduces the inertia of the system, thus increasing the risk of instability when disturbance occurs. The VSG produces virtual inertia by introducing the dynamic characteristics of the synchronous generator, which provides inertia and becomes a grid-forming control method. The disadvantages of the VSG are that there are many parameters to be adjusted and its operation process is complicated. However, with the rapid development of artificial intelligence (AI) technology, the powerful adaptive learning capability of AI algorithms provides potential solutions to this issue. Two research hotspots are deep learning (DL) and reinforcement learning (RL). This paper presents a comprehensive review of these two techniques combined with VSG control in the energy internet (EI). Firstly, the basic principle and classification of the VSG are introduced. Next, the development of DL and RL algorithms is briefly reviewed. Then, recent research on VSG control based on DL and RL algorithms are summarized. Finally, some main challenges and study trends are discussed. [ABSTRACT FROM AUTHOR]

The integration of electric vehicles (EVs) into vehicle-to-grid (V2G) scheduling offers a promising opportunity to enhance the profitability of multi-energy microgrid operators (MMOs). MMOs aim to maximize their total profits by coordinating V2G scheduling and multi-energy flexible loads of end-users while adhering to operational constraints. However, scheduling V2G strategies online poses challenges due to uncertainties such as electricity prices and EV arrival/departure patterns. To address this, we propose an online V2G scheduling framework based on deep reinforcement learning (DRL) to optimize EV battery utilization in microgrids with different energy sources. Firstly, our approach proposes an online scheduling model that integrates the management of V2G and multi-energy flexible demands, modeled as a Markov Decision Process (MDP) with an unknown transition. Secondly, a DRL-based Soft Actor-Critic (SAC) algorithm is utilized to efficiently train neural networks and dynamically schedule EV charging and discharging activities in response to real-time grid conditions and energy demand patterns. Extensive simulations are conducted in case studies to testify to the effectiveness of our proposed approach. The overall results validate the efficacy of the DRL-based online V2G scheduling framework, highlighting its potential to drive profitability and sustainability in multi-energy microgrid operations. [ABSTRACT FROM AUTHOR]

The task scheduling of e-government cloud center has always been a complex problem. Most existing task scheduling solutions rely on expert knowledge and are not versatile enough to deal with dynamic cloud environment, which often leads to low resource utilization and degradation of quality-of-service, resulting in longer makespan. To address this issue, this paper proposed a deep reinforcement learning(DRL) scheduling algorithm based on the actor-critic(A2C) mechanism. Firstly, the actor network parameterized the policy and chose scheduling actions based on the current system state, while the critic network assigned scores to the current system state. Then, it updated the actor policy network using gradient ascent, utilizing the scores from the critic network to determine the effectiveness of actions. Finally, it conducted simulation experiments using real data from production datacenters. The results show that this method can improve resource utilization in cloud datacenters and reduce the makespan in comparison to the classic policy gradient algorithm and five commonly used heuristic task scheduling methods. This evidence suggests that the proposed method is superiorly adapted for the dynamic e-government clouds. [ABSTRACT FROM AUTHOR]

This paper aimed to optimize the node selection problem in existing oracle machine schemes in the blockchain environment in order to improve the accuracy and reliability of oracle machine node selection. This paper introduced ORLM, a deep reinforcement learning based middleware for blockchain prophecy machine node selection. This middleware considered the consumption of multiple nodes under different service demands and modeled the reputation value of the prophecy machine nodes to sick the reputation value of the oracle machine data-providing nodes, thus avoiding the selection of nodes with malicious history as much as possible. With the deep reinforcement learning DQN algorithm, the middleware is able to optimize the process of selecting nodes for better node selection with security. The experimental results show that the oracle middleware is able to better satisfy the service requests of users. And it has high scalability and availability. It proves that the introduction of deep reinforcement learning to optimize the oracle node selection is a feasible direction. [ABSTRACT FROM AUTHOR]

This paper proposes a method for finding the shortest path of a mobile robot using deep reinforcement learning with utilizing Proximal policy optimization algorithm (PPO) enhanced with curriculum learning. By modelling the environment in 3D space using the Webots simulator, we extend the PPO algorithm's capabilities to handle continuous states from 8 IR sensors and control the velocities of two motors of E-puck robot. Our study uniquely integrates curriculum learning into the PPO framework, aiming to improve adaptability and training efficiency in complex environments. A comparative analysis is conducted between the modified PPO, the original PPO, and the deep deterministic policy gradient algorithm, highlighting the strengths of our approach The results demonstrate that our curriculum-augmented PPO algorithm not only accelerates the training process but also shows superior adaptability and generalization in new environments. This work underscores the significant potential of curriculum learning in enhancing the performance of deep reinforcement learning algorithms for robust and efficient robotic navigation. [ABSTRACT FROM AUTHOR]

Cooling systems provide a safe thermal environment for the reliable operation of IT equipment in data centers (DCs) while generating significant energy consumption. Therefore, to achieve energy savings in cooling system control under dynamic thermal distribution in DCs, this paper proposes a multi-setpoint cooling control approach based on deep reinforcement learning (DRL). Firstly, a thermal model based on the XGBoost algorithm is constructed to precisely evaluate the thermal distribution in the rack room to guide real-time cooling control. Secondly, a multi-set point cooling control approach based on the deep Q-network algorithm (DQN-MSP) is designed to finely regulate the supply air temperature of each air conditioner by capturing the thermal fluctuations to ensure the dynamic balance of cooling supply and demand. Finally, we adopt the extended CloudSimPy simulation tool and the real workload trace of the PlanetLab system to evaluate the effectiveness and performance of the proposed approach. The simulation results show that the proposed control solution effectively reduces the cooling energy consumption by over 2.4% by raising the average air supply temperature of the air conditioner while satisfying the thermal constraints. [ABSTRACT FROM AUTHOR]

Background: Volumetric modulated arc therapy (VMAT) machine parameter optimization (MPO) remains computationally expensive and sensitive to input dose objectives creating challenges for manual and automatic planning. Reinforcement learning (RL) involves machine learning through extensive trial‐and‐error, demonstrating performance exceeding humans, and existing algorithms in several domains. Purpose: To develop and evaluate an RL approach for VMAT MPO for localized prostate cancer to rapidly and automatically generate deliverable VMAT plans for a clinical linear accelerator (linac) and compare resultant dosimetry to clinical plans. Methods: We extended our previous RL approach to enable VMAT MPO of a 3D beam model for a clinical linac through a policy network. It accepts an input state describing the current control point and predicts continuous machine parameters for the next control point, which are used to update the input state, repeating until plan termination. RL training was conducted to minimize a dose‐based cost function for prescription of 60 Gy in 20 fractions using CT scans and contours from 136 retrospective localized prostate cancer patients, 20 of which had existing plans used to initialize training. Data augmentation was employed to mitigate over‐fitting, and parameter exploration was achieved using Gaussian perturbations. Following training, RL VMAT was applied to an independent cohort of 15 patients, and the resultant dosimetry was compared to clinical plans. We also combined the RL approach with our clinical treatment planning system (TPS) to automate final plan refinement, and creating the potential for manual review and edits as required for clinical use. Results: RL training was conducted for 5000 iterations, producing 40 000 plans during exploration. Mean ± SD execution time to produce deliverable VMAT plans in the test cohort was 3.3 ± 0.5 s which were automatically refined in the TPS taking an additional 77.4 ± 5.8 s. When normalized to provide equivalent target coverage, the RL+TPS plans provided a similar mean ± SD overall maximum dose of 63.2 ± 0.6 Gy and a lower mean rectum dose of 17.4 ± 7.4 compared to 63.9 ± 1.5 Gy (p = 0.061) and 21.0 ± 6.0 (p = 0.024) for the clinical plans. Conclusions: An approach for VMAT MPO using RL for a clinical linac model was developed and applied to automatically generate deliverable plans for localized prostate cancer patients, and when combined with the clinical TPS shows potential to rapidly generate high‐quality plans. The RL VMAT approach shows promise to discover advanced linac control policies through trial‐and‐error, and algorithm limitations and future directions are identified and discussed. [ABSTRACT FROM AUTHOR]

A strategy refers to the rules that the agent chooses the available actions to achieve goals. Adopting reasonable strategies is challenging but crucial for an intelligent agent with limited resources working in hazardous, unstructured, and dynamic environments to improve the system's utility, decrease the overall cost, and increase mission success probability. This article proposes a novel hierarchical strategy decomposition approach based on Bayesian chaining to separate an intricate policy into several simple sub-policies and organize their relationships as Bayesian strategy networks (BSN). We integrate this approach into the state-of-the-art DRL method—soft actor-critic (SAC), and build the corresponding Bayesian soft actor-critic (BSAC) model by organizing several sub-policies as a joint policy. Our method achieves the state-of-the-art performance on the standard continuous control benchmarks in the OpenAI Gym environment. The results demonstrate that the promising potential of the BSAC method significantly improves training efficiency. Furthermore, we extend the topic to the Multi-Agent systems (MAS), discussing the potential research fields and directions. [ABSTRACT FROM AUTHOR]

This paper proposes a novel cloud computingbased parallel deep reinforcement learning (DRL) energy management strategy (EMS) for connected plug-in hybrid vehicles. First, a proximal policy optimization (PPO) algorithm is developed. Since the cloud computing can reduce the computational burden of the connected vehicles, the PPO is deployed in the proposed cloud computing-based EMS. In order to improve the strategy adaptation, a parallel mechanism is proposed to achieve the information interaction with multiple vehicles. Considering the real-time control requirements, thread pool is proposed and applied in the cloud computing based parallel EMS. The thread pool-based strategy provides an efficient real-time control ability and strategy improvement solution. To verify the PPO based EMS, the dynamic programming, deep Q-network and double deep Q-network strategies are developed for comparison. It is found that the PPO can achieve similar fuel efficiency improvement with the DP strategy among the three DRL algorithms. For parallel training of multiple connected vehicles, the cloud computing-based parallel EMS improves fuel economy by approximately 7.7%. Threadpool based parallel real-time EMS reduces average time for computational interactions by 20% and further improves the fuel efficiency. The proposed strategy has the advantages of realtime control, adaptability and continuous learning for improved fuel efficiency. [ABSTRACT FROM AUTHOR]

The frequently evolving manufacturing system necessitates real-time large-scale data analytics, surpassing the capabilities of classical systems or human skills, necessitating knowledge development and self-adaptive control. Unpredictable real-time events in smart factories cause changes in the effectiveness of planned schedules or tasks; even slight interruptions can accumulate to make the preplanned schedule unoptimized, if not impossible. To meet the required production efficiency, typical dynamic scenarios on the shop floor, such as failures, random job arrivals, and machine setup, must be addressed quickly. Hence, this study proposes a novel Triple Deep Q Network (TDQN) approach for learning high-quality dispatching rules for addressing the Flexible Job Shop Scheduling Problem (FJSSP) in uncertainty. For the dispatching rule-based FJSSP, the Markov Decision Process (MDP) to choose a suitable operation-machine (O-M) pair is formulated to allow operation selection and resource assignment resolutions to be made simultaneously. The performance of the TDQN method is investigated against the Double Deep Q Network (DDQN) and Deep Q Network (DQN) approach and found to be more stable. Moreover, among the 29 different shopfloor configuration settings, TDQN performed better in 62.07% of the overall instances, while DDQN and DQN performed better in only 31.03% and 6.9% of instances, respectively. [ABSTRACT FROM AUTHOR]

The exposure of zero trust security in the Industrial Internet of Things (IIoT) increased in importance in the era where there is a huge risk of injection of malicious entities and owning the device by an unauthorized user. The gap in the existing approach of zero trust security is that continuous verification of devices is a time-consuming process and adversely affects the promising nature of the zero-trust model. Every time the node enters, even if the node is a member of the network, authorization of the node is necessary to ensure authenticity. This verification section of zero trust hinders the seamless working of the IIoT infrastructure. Therefore, the main objective of this paper is to propose the solution for the above-mentioned problem by enabling "device profiling" via deep reinforcement learning so that the same device can be identified and permitted access without hindering the working of Industrial Internet of Things infrastructure. The overall proposed approach works in different phases including the compression function for ensuring data confidentiality and integrity, then the device profiling is performed based on the features a device possesses, and lastly, deep reinforcement learning for anomaly detection. To test and validate the proposed approach, extensive experimentations were performed using measures such as false positive rate, data confidentiality rate, data integrity rate, and network access time, and results showed that the proposed technique titled "MMODPAD-DRL" outperforms the existing approaches in false positive rate by 27%, data confidentiality rate by 4% and data integrity rate by 3%, in addition, lessen the network access time by 20%. [ABSTRACT FROM AUTHOR]

In recent years, space research has shifted heavily its focus towards enhanced autonomy on-board spacecrafts for on-orbit servicing activities (OOS). OOS and proximity operations include a variety of activities: the focal point of this work is the autonomous guidance of a chaser spacecraft for the shape reconstruction of an artificial uncooperative object. Adaptive guidance depends on the ability of the system to build a map of the uncertain environment, figuring out its location inside of it and accordingly determining the control law. Thus, autonomous navigation is framed as an active Simultaneous Localization and Mapping (SLAM) problem and modeled as a Partially Observable Markov Decision Process (POMDP). A state-of-the-art Deep Reinforcement Learning (DRL) method, Proximal Policy Optimization (PPO), is investigated to develop an agent capable of cleverly planning the shape reconstruction of the target. Starting from previous research on the topic, this work develops further proposing a continuous action space, such that the agent is no more forced to choose between a predefined set of possible discrete actions, fixed both in magnitude and direction. In this way any combination of the three-dimensional thrust vector components is available. The chaser spacecraft is a small satellite mounting an electric propulsion engine defining the action space range, in linearized eccentric relative motion with the selected uncooperative object. Through a rendered triangular mesh, the agent capabilities of geometry reconstruction and mapping are evaluated, considering the number of quality pictures made for each face. Extensive training tests are performed with random initial conditions to verify the generalizing capability of the DRL agent. The results are then validated in a comprehensive testing campaign, whose primary focus is the introduction of noisy measurements coming from navigation, affecting pose estimation. The sensitivity of the proposed method to this condition is analyzed and the efficiency of a retraining procedure is examined. The applicability of DRL methods and neural networks to support autonomous guidance in a close proximity scenario is corroborated and the technique employed is vastly tested and verified. [ABSTRACT FROM AUTHOR]

Control–structure interaction (CSI) plays a significant role in active control systems. Popular methods incorporate actuator dynamics into an integrated control system to account for CSI, leading to a situation where existing structural control algorithms that ignore CSI cannot be applied directly. To address this issue, this study proposes a deep reinforcement learning (DRL) based active mass driver (AMD) decoupled control framework, in which a structural control algorithm is employed to generate the control force command without consideration of CSI, while a DRL agent is utilized to attenuate the CSI effects of AMD systems and achieve a desired control force. The DRL‐based AMD control framework is verified through a series of numerical experiments. Furthermore, the applicability of the control framework is confirmed in a wind‐excited 76‐story benchmark building. Comprehensive analysis indicates that the proposed control framework can effectively eliminate the CSI effects and apply accurate control force to the structure in various scenarios, which allows for an ideal structural response control. [ABSTRACT FROM AUTHOR]

• Performance of object detection tends to degrade on low-quality images. • Choosing suitable enhancement algorithms can improve the detection performance. • Reinforcement learning can autonomously select algorithms based on the scenario. • Reinforcement learning avoid the issue of data labeling and retraining. The task of object detection for low-quality images in multiple scenarios remains a challenging task. To alleviate this problem, a novel approach to improve the detection performance of the model on low-quality images using deep reinforcement learning is proposed. The approach differs from methods that involve data labeling and retraining, as the detection results of the model are utilized to formulate a reward function. This enables deep reinforcement learning to transform low-quality images into high-quality images through image enhancement, ultimately leading to an improvement in the model's recognition performance on low-quality images. An image enhancement tool chain (IETC) has been prepared, offering a flexible and lightweight algorithm selection for the Dueling Deep Q-Network-based Tool Selector (DDQN-TS). By adding a "Pass" option, Enable DDQN-TS to learn the relative relationship between performing and not performing image enhancement. To mitigate the potential inconsistency between the output of the detection model and the actual situation caused by negative samples, which can lead to incorrect decisions made by DDQN-TS, the introduction of a "thresh" parameter regulates the level of intervention in target detection by DIE. The experimental results demonstrate the effectiveness and portability of the DIE model in environments with noise, fog, and uneven brightness. [ABSTRACT FROM AUTHOR]

In the harsh mobile edge computing (HMEC) environment, there are many dynamic changes such as interference from noise, the impact of extreme environmental conditions, and the mobility of devices. It is a great challenge to the online realtime task offloading scheduling for delay-sensitive applications. However, the dynamic changes in HMEC environment have been ignored in almost all previous studies. Therefore, we propose the federated deep reinforcement learning-based online task offloading and resource allocation (FD-OTR) algorithm to address the task offloading in HMEC. Additionally, the FD-OTR algorithm performs resource allocation for offloaded tasks. The task offloading part of FD-OTR algorithm can be divided into two layers: the deep reinforcement learning (DRL) layer and the federated learning (FL) layer. The online algorithm in the DRL layer can adapt to the dynamic HMEC environment and make real-time task offloading decisions. In the FL layer, federated learning with low communication overhead is used for model aggregation to form a better global model. Resource allocation is done by using a new meta-heuristic algorithm: the Sparrow Search Algorithm (SSA). Finally, the simulation results demonstrate that the FD-OTR algorithm performs well in HMEC. The convergence speed of FD-OTR is three times faster than the centralized method. Compared to the baseline algorithms, FD-OTR reduces costs by 14.3%, 11.2% and 9.28%, respectively. [ABSTRACT FROM AUTHOR]

The advent of Industry 4.0 has significantly promoted the field of intelligent manufacturing, which is facilitated by the development of new technologies are emerging. Robot technology and robot intelligence methods have rapidly developed and been widely applied. Manipulators are widely used in industry, and their control is a crucial research topic. The inverse kinematics solution of manipulators is an important part of manipulator control, which calculates the joint angles required for the end effector to reach a desired position and posture. Traditional inverse kinematics solution algorithms often face the problem of insufficient generalization, and iterative methods have challenges such as large computation and long solution time. This paper proposes a reinforcement learning-based inverse kinematics solution algorithm, called the MAPPO-IK algorithm. The algorithm trains the manipulator agent using the MAPPO algorithm and calculates the difference between the end effector state of the manipulator and the target posture in real-time by designing a reward mechanism, while considering Gaussian distance and cosine distance. Through experimental comparative analysis, the feasibility, computational efficiency, and superiority of this reinforcement learning algorithm are verified. Compared with traditional inverse kinematics solution algorithms, this method has good generalization and supports real-time computation, and the obtained result is a unique solution. Reinforcement learning algorithms have better adaptability to complex environments and can handle different sudden situations in different environments. This algorithm also has the advantages of path planning, intelligent obstacle avoidance, and other advantages in dynamically processing complex environmental scenes. [ABSTRACT FROM AUTHOR]

With the widespread adoption of electric vehicles (EVs), the demand for public charging services is steadily increasing. Consequently, the development of effective charging scheduling strategies, aimed at optimizing the utilization of limited charging infrastructure, has become a key problem. Considering the diversity of user demands, we propose a Cost-Aware Charging Scheduling Architecture (CASA). This architecture considers both urgent and nonurgent charging customers by designing two charging modes with different power levels and associated costs. However, optimizing multiple objectives simultaneously while ensuring the interests of all parties involved in the charging demand response presents a challenge. Moreover, the uncertainty in customer charging demands and Time-of-Use (TOU) tariff further complicates the establishment of the model. To address the aforementioned challenges, this study formulates EV charging scheduling as a Markov Decision Process (MDP) based on deep reinforcement learning (DRL), employing the Deep Q-Network (DQN) algorithm for solution derivation. The objective is to minimize the operational costs of charging stations while ensuring the quality of service (QoS) requirements for customers. The simulation results demonstrate that CASA exhibits superior performance in optimizing both the average response time and service success rate, compared to commonly used baselines for charging scheduling. Furthermore, the CASA approach achieves a significant reduction in operating costs of EV charging station. [ABSTRACT FROM AUTHOR]

As the bias between the estimated value and the true value, overestimation is a basic problem in reinforcement learning, which leads to a lower total reward because of the incorrect action decisions. In order to reduce the impact of overestimation on reinforcement learning, we propose a double Actor-Critic learning system embedding improved Monte Carlo Tree Search (DAC-IMCTS). The proposed learning system consists of a reference module, a simulation module and an outcome module. The reference module and the simulation module are designed to compute the upper bound and lower bound of the expected reward of the agent, respectively. And the outcome module is developed to learn the agent's control policies. The reference module, constructed based on the Actor-Critic framework, provides an upper confidence bound of the expected reward. Different from the classic Actor-Critic learning system, we introduce a simulation module into the new learning system to estimate the lower confidence bound of the expected reward. We propose an improved MCTS in this module to sample the policy distribution more efficiency. Based on the lower and upper confidence bounds, we propose a confidence interval weighted estimation algorithm (CIWE) in the outcome module for generating the target expected reward. We then prove that the target expected reward generated by our method has zero expectation bias, which reduces the overestimation that exists in the classic Actor-Critic learning system. We evaluate our learning system on OpenAI Gym experimental tasks. The experimental results show that our proposed model and algorithm outperform the state-of-the-art learning systems. [ABSTRACT FROM AUTHOR]

The unmanned combat system-of-systems (UCSoS) in modern warfare is comprised of various interconnected entities that work together to support mission accomplishment. The soaring number of entities makes the UCSoS fragile and susceptible to triggering cascading effects when exposed to uncertain disturbances such as attacks or failures. Reconfiguring the UCSoS to restore its effectiveness in a self-coordinated and adaptive manner based on the battlefield situation and operational requirements has attracted increasing attention. In this paper, we focus on the UCSoS reconstruction with heterogeneous costs, where the collaboration nodes may have different reconstruction costs. Specifically, we adopt the heterogeneous network to capture the interdependencies among combat entities and propose a more representative metric to evaluate the UCSoS reconstruction effectiveness. Next, we model the combat network reconstruction problem with heterogeneous costs as a nonlinear optimization problem and prove its NP-hardness. Then, we propose an approach called SoS-Restorer, which is based on deep reinforcement learning (DRL), to address the UCSoS reconstruction problem. The results show that SoS-Restorer can quickly generate reconstruction strategies and improve the operational capabilities of the UCSoS by about 20∼60% compared to the baseline algorithm. Furthermore, even when the size of the UCSoS exceeds that of the training data, SoS-Restorer exhibits robust generalization capability and can efficiently produce satisfactory results in real time. [ABSTRACT FROM AUTHOR]

Vehicular Edge Computing (VEC) is a crucial application of Mobile Edge Computing (MEC) in vehicular networks. In VEC networks, the computation tasks of vehicle terminals (VTs) can be offloaded to nearby MEC servers, overcoming the limitations of VTs' processing power and reducing latency caused by distant cloud communication. However, a mismatch between VTs' demanding tasks and MEC servers' limited resources can overload MEC servers, impacting Quality of Service (QoS) for computationally intensive tasks. Additionally, vehicle mobility can disrupt communication with static MEC servers, further affecting VTs' QoS. To address these challenges, this paper proposes a vehicle collaborative partial computation offloading model. This model allows VTs to offload tasks to two types of service nodes: collaborative vehicles and MEC servers. Factors like a vehicle's mobility, remaining battery power, and available computational power are also considered when evaluating its suitability for collaborative offloading. Furthermore, we design a deep reinforcement learning-based strategy for collaborative partial computation offloading that minimizes overall task delay while meeting individual latency constraints. Experimental results demonstrate that compared to traditional approaches without vehicle collaboration, this scheme significantly reduces latency and achieves a significant reduction (around 2%) in the failure rate under tighter latency constraints. [ABSTRACT FROM AUTHOR]

The IEEE 802.11ah standard is introduced to address the growing scale of internet of things (IoT) applications. To reduce contention and enhance energy efficiency in the system, the restricted access window (RAW) mechanism is introduced in the medium access control (MAC) layer to manage the significant number of stations accessing the network. However, to achieve optimized network performance, it is necessary to appropriately determine the RAW parameters, including the number of RAW groups, the number of slots in each RAW, and the duration of each slot. In this paper, we optimize the configuration of RAW parameters in the uplink IEEE 802.11ah-based IoT network. To improve network throughput, we analyze and establish a RAW parameters optimization problem. To effectively cope with the complex and dynamic network conditions, we propose a deep reinforcement learning (DRL) approach to determine the preferable RAW parameters to optimize network throughput. To enhance learning efficiency and stability, we employ the proximal policy optimization (PPO) algorithm. We construct network environments with periodic and random traffic in an NS-3 simulator to validate the performance of the proposed PPO-based RAW parameters optimization algorithm. The simulation results reveal that using the PPO-based DRL algorithm, optimized RAW parameters can be obtained under different network conditions, and network throughput can be improved significantly. [ABSTRACT FROM AUTHOR]

Recently, unmanned aerial vehicles (UAVs) have been widely used in mobile edge computing (MEC) scenarios due to their flexibility, rapid deployment, and ability to expand communication coverage. This paper explores how UAVs can compute offloading more effectively when serving as mobile edge servers in the scenario of UAVs-assisted MEC. We aim to maximize the utility of the UAV operator by joint flight trajectory of UAVs, users' offloading decisions, and collaborative strategy among UAVs while considering the latency and energy consumption. To this end, we first propose a dynamic latency-aware collaborative framework based on deep reinforcement learning. This framework utilizes a queuing model to implement the dynamic task backlog and models the reward of latency-sensitive tasks using a service level agreement. Then, considering the dynamics of the environment and the complexity of the problem, we describe the problem as a markov decision process and propose a proximal policy optimization-based computation offloading algorithm to solve the problem. The experimental results finally demonstrate that the proposed offloading algorithm converges rapidly and exhibits better performance compared to benchmark algorithms. [ABSTRACT FROM AUTHOR]

To address the current challenge of smart devices in healthcare Internet of things (IoT) struggling to efficiently process intensive applications in real-time, a collaborative cloud-edge offloading model tailored for ultra-dense edge computing (UDEC) networks is developed. While numerous studies have delved into the optimization of offloading in mobile edge computing (MEC), it is imperative to consider non-orthogonal multiple access (NOMA) as a physical technology when addressing the offloading optimization process in MEC. The multiuser sharing of spectrum resources in NOMA can enhance the network spectrum utilization and reduce the computational delay when users transmit computing tasks. Consequently, a model for NOMA-assisted UDEC systems is proposed. The model takes into account joint offloading decisions, computational resources, and sub-channel resources and is modeled as a complex nonlinear mixed-integer programming problem. The aim is to decrease the task execution delay and energy consumption of smart devices while ensuring that users' maximum acceptable delay for processing medical computational tasks is met efficiently and in a timely manner. Deep deterministic policy gradient (DDPG), a deep reinforcement learning method, is employed to solve the joint optimization problem. The final simulation results show that the algorithm converges well. The proposed offloading scheme can reduce the system cost by 54.5 and 69.9% in comparison with scenarios where users solely perform local computations and offload their tasks to the base station (BS). The application of NOMA communication in our offloading scheme boosts network spectrum utilization and trims down the system cost by 87.09% when contrasted with orthogonal multiple access (OMA). [ABSTRACT FROM AUTHOR]

Summary: Cognitive radio networks (CRNs) have emerged as a promising solution to address the spectrum scarcity problem by allowing unlicensed secondary users (SUs) to opportunistically access the underutilized licensed spectrum bands. However, efficient channel assignment in CRNs, especially with user mobility and power control considerations, remains a challenging problem. Traditional channel assignment algorithms often struggle to adapt to changing network conditions, resulting in suboptimal performance. To address these challenges, this research presents a novel Taylor Tuna Optimizer‐based hybrid deep Q‐network (TayTO‐based HDQNet) for channel assignment in CRNs with user mobility and power control considerations. The proposed TayTO‐based HDQNet combines the Elman neural network architecture with the reinforcement learning framework of deep Q‐networks, tuned using the Taylor Tuna Optimizer. This hybrid approach enables intelligent channel assignment decisions while considering power consumption constraints, leading to efficient communication in CRNs. To train the HDQNet agent, a comprehensive dataset is generated through simulations of various CRN scenarios, incorporating channel availability, interference levels, and power consumption for different channel assignment decisions. The HDQNet agent undergoes iterative training using this dataset to develop an optimal channel assignment policy. Experimental results demonstrate the effectiveness of the proposed TayTO‐based HDQNet approach done by MATLAB, achieving a high success rate of 98.47% and surpassing the reliability scores of previous studies. This highlights the improved performance and reliability of the proposed approach for channel assignment in CRNs. [ABSTRACT FROM AUTHOR]

The increasing prevalence of Internet of Things (IoT) systems has made them attractive targets for malicious actors. To address the evolving threats and the growing complexity of detection, there is a critical need to search for and develop new algorithms that are fast and robust in detecting and classifying dangerous network traffic. In this context, deep reinforcement learning (DRL) is gaining recognition as a prospective solution in numerous fields as it enables autonomous agents to cooperate with their environment for decision-making without relying on human experts. This article presents an innovative approach to intrusion detection in IoT systems using an adversarial reinforcement learning (RL) algorithm known for its exceptional predictive capabilities. The predictive process relies on a classifier, implemented as a streamlined and highly efficient neural network. Embedded within this classifier is a policy function meticulously trained using an innovative RL model. Importantly, this model ensures that the environment's behavior is dynamically fine-tuned simultaneously with the learning process, improving the overall effectiveness of the intrusion detection approach. The efficiency of our proposal was assessed using the Bot-IoT database, consisting of a mixture of legitimate IoT network traffic and simulated attack scenarios. Our scheme shows superior performance compared to existing ones. Therefore, our approach to IoT intrusion detection can be considered a valuable alternative to existing methods, capable of significantly improving the IoT systems' security. [ABSTRACT FROM AUTHOR]

Navigation of a mobile robot in an unknown environment ensuring the safety of the robot and its surroundings is of utmost importance. Traditional methods, such as pathplanning algorithms, simultaneous localization and mapping, computer vision, and fuzzy techniques, have been employed to address this challenge. However, to achieve better generalization and self-improvement capabilities, reinforcement learning has gained significant attention. The concern of privacy issues in sharing data is also rising in various domains. In this study, a deep reinforcement learning strategy is applied to the mobile robot to move from its initial position to a destination. Specifically, the Deep Q-Learning algorithm has been used for this purpose. This strategy is trained using a federated learning approach to overcome privacy issues and to set a foundation for further analysis of distributed learning. The application scenario considered in this work involves the navigation of a mobile robot to a charging point within a greenhouse environment. The results obtained indicate that both the traditional deep reinforcement learning and federated deep reinforcement learning frameworks are providing 100% success rate. However federated deep reinforcement learning could be a better alternate since it overcomes the privacy issue along with other advantages discussed in this paper. [ABSTRACT FROM AUTHOR]

Meta-learning is a pivotal and potentially influential machine learning approach to solve challenging problems in reinforcement learning. However, the costly hyper-parameter tuning for training stability of meta-learning is a known shortcoming and currently a hotspot of research. This paper addresses this shortcoming by introducing an online and easily trainable hyper-parameter optimization approach, called Meta Parameters Learning via Meta-Learning (MPML), to combine online hyper-parameter adjustment scheme into meta-learning algorithm, which reduces the need to tune hyper-parameters. Specifically, a basic learning rate for each training task is put forward. Besides, the proposed algorithm dynamically adapts multiple basic learning rate and a shared meta-learning rate through conducting gradient descent alongside the initial optimization steps. In addition, the sensitivity with respect to hyper-parameter choices in the proposed approach are also discussed compared with model-agnostic meta-learning method. The experimental results on reinforcement learning problems demonstrate MPML algorithm is easy to implement and delivers more highly competitive performance than existing meta-learning methods on a diverse set of challenging control tasks. [ABSTRACT FROM AUTHOR]

Variational quantum circuits are one of the promising ways to exploit the advantages of quantum computing in the noisy intermediate-scale quantum technology era. The design of the quantum circuit architecture might greatly affect the performance capability of the quantum algorithms. The quantum architecture search is the process of automatically designing quantum circuit architecture, aiming at finding the optimal quantum circuit composition architecture by the algorithm for a given task, so that the algorithm can learn to design the circuit architecture. Compared to manual design, quantum architecture search algorithms are more effective in finding quantum circuits with better performance capabilities. In this paper, based on the deep reinforcement learning, we propose an approach for quantum circuit architecture search. The sampling of the circuit architecture is learnt through reinforcement learning-based controller. Layer-based search is also used to accelerate the computational efficiency of the search algorithm. Applying to data classification tasks, we show that the method can search for quantum circuit architectures with better accuracies. Moreover, the circuit has a smaller number of quantum gates and parameters. [ABSTRACT FROM AUTHOR]

As wireless networks continue to advance, virtual reality (VR) transmission over wireless connections is progressively transitioning from concept to practical application. Although this technology can significantly enhance the VR user experience, its development bottleneck lies in the computing capacity of devices and transmission latency. Considering the limited computational resources of VR devices for rendering tasks, multi-access edge computing (MEC) servers are introduced to provide powerful computing capabilities. To cope with transmission latency, reconfigurable intelligent surface (RIS) enhances links between base stations (BSs) and users. Based on these two technologies, we propose a RIS-assisted VR streaming model, where BSs are equipped with MEC servers to assist data rendering. Firstly, the user association, power control, and RIS phase shift optimization problems in the VR transmission system are jointly modeled and analyzed, establishing a long-term minimization of the interaction delay model. Secondly, by modeling the optimization problem as a Markov decision process (MDP), a joint optimization framework based on multi-agent deep reinforcement learning (MADRL) is proposed. In this framework, we have separately designed two dedicated algorithms for discrete and continuous variables. Furthermore, multiple agents can provide feedback based on user experience and cooperate with each other to improve the joint strategy. Finally, the performance and superiority of the proposed solution and algorithm are validated through simulation experiments in different application scenarios. [ABSTRACT FROM AUTHOR]

The present study applies a Deep Reinforcement Learning (DRL) algorithm to Active Flow Control (AFC) of a two-dimensional flow around a confined square cylinder. Specifically, the Soft Actor-Critic (SAC) algorithm is employed to modulate the flow of a pair of synthetic jets placed on the upper and lower surfaces of the confined squared cylinder in flow configurations characterized by Re of 100, 200, 300, and 400. The investigation starts with an analysis of the baseline flow in the absence of active control. It is observed that at Re = 100 and Re = 200, the vortex shedding exhibits mono-frequency characteristics. Conversely, at Re = 300 and Re = 400, the vortex shedding is dominated by multiple frequencies, which is indicative of more complex flow features. With the application of the SAC algorithm, we demonstrate the capability of DRL-based control in effectively suppressing vortex shedding, while significantly diminishing drag and fluctuations in lift. Quantitatively, the data-driven active control strategy results in a drag reduction of approximately 14.4%, 26.4%, 38.9%, and 47.0% for Re = 100, 200, 300, and 400, respectively. To understand the underlying control mechanism, we also present detailed flow field comparisons, which showcase the adaptability of DRL in devising distinct control strategies tailored to the dynamic conditions at varying Re. These findings substantiate the ability of DRL to control chaotic, multi-frequency dominated vortex shedding phenomena, underscoring the robustness of DRL in complex AFC problems. [ABSTRACT FROM AUTHOR]

Addressing the exploration challenges faced by mobile robots using deep reinforcement learning for autonomous navigation in cluttered, obstacle-dense complex environments, this paper proposed the trajectory-guided navigation policy optimization (TGNPO) algorithm. Firstly, it employed an imitation learning approach to train an expert policy for a mobile robot, which could provide both expert demonstration behavior and navigation trajectory prediction and aimed to comprehensively guide the training of deep reinforcement learning. Secondly, it fused the predicted navigation trajectory from the expert policy with real-time images perceived by the mobile robot at the current moment. Combining the coordinate attention mechanism, it extracted feature regions which would guide the robot’s future navigation, thereby enhancing the learning performance of the navigation model. Finally, it utilized the navigation trajectory predicted by the expert policy to constrain the policy trajectory of the mobile robot, mitigating ineffective exploration and erroneous decision-making during navigation process By deploying the proposed algorithm on both simulation and physical platforms, experimental results demonstrated significant advantages in navigation learning efficiency and trajectory smoothness compared to existing state-of-the-art methods which fully proves the proposed algorithm’s capability to efficiently and safely execute robot navigation tasks. [ABSTRACT FROM AUTHOR]

Multi-agent systems have a wide range of applications in many fields, such as autonomous driving, intelligent logistics, and medical collaboration, etc. However, due to technological advances and increased system requirements, these systems face challenges such as large scale and high complexity, and often suffer from inefficient training and poor adaptability. To address these problems, this paper proposed a multi-agent first-order meta proximal policy optimization (MAMPPO) method by extending gradient-based meta-learning to multi-agent deep reinforcement learning. The method learned the initial model parameters in the multi-agent system to provide a new perspective for improving the performance of multi-agent deep reinforcement learning. It made full use of the previous experience in the process of multi-agent reinforcement learning to find the most sensitive parameters in the direction of gradient descent through repeated adaptation, and learned the initial parameters so that the model training starts from the optimal starting point. This method effectively improved the decision-making efficiency of the joint policy, and led to a significant increase in the speed of its policy change, which significantly accelerated the speed of adaptation in the face of a new situation. Experimental results on StarCraft Ⅱ show that the MAMPPO method can significantly improve the training speed and adaptability, which provides a new solution for the subsequent improvement of the training efficiency and adaptability of multi-agent reinforcement learning. [ABSTRACT FROM AUTHOR]

Optimization of industrial processes such as manufacturing or processing of specific materials constitutes a point of interest for many researchers, and its application can lead not only to speeding up the processes in question, but also to reducing the energy cost incurred during them. This article presents a novel approach to optimizing the spindle motion of a computer numeric control (CNC) machine. The proposed solution is to use deep learning with reinforcement to map the performance of the reference points realization optimization (RPRO) algorithm used in the industry. A detailed study was conducted to see how well the proposed method performs the targeted task. In addition, the influence of a number of different factors and hyperparameters of the learning process on the performance of the trained agent was investigated. The proposed solution achieved very good results, not only satisfactorily replicating the performance of the benchmark algorithm, but also speeding up the machining process and providing significantly higher accuracy. [ABSTRACT FROM AUTHOR]

Deep reinforcement learning has proven to be effective in various video games, such as Atari games, StarCraft II, Google research football (GRF), and Dota II. We participated in the 2022 IEEE Conference on Games Football AI Competition and ranked in the top eight. Despite recent efforts, building agents for GRF still suffers from multi-agent coordination, sparse rewards, and stochastic environments. To address these issues and achieve good outcomes in the competition, we devised a reinforcement learning algorithm that uses deep reinforcement learning from demonstrations and policy distillation. In this study, we innovatively propose a two-stage algorithm named mimic-to-counteract reinforcement learning (MCRL) based on the historical game logs of opponents, we encountered during the warm-up session and formulated partner agents function similarly to human sparring partners, whereby they simulate opponents with diverse styles of play, enabling primary players to practice against a range of policies, they may encounter in real competitions. Additionally, we trained numerous mentor agents capable of restraining the sparring partners. We distilled their policies and amalgamated them to train a potent primary agent. Empirical results show that the proposed MCRL algorithm can efficiently search for valuable strategies with stable updates and balance the relationship between policy iteration and policy style deviation. Also, the primary agent can learn diverse but coordinated counteracting strategies and ranks in the top eight in the competition. [ABSTRACT FROM AUTHOR]

In this paper, we consider a low-latency Mobile Edge Computing (MEC) network where multiple User Equipment (UE) wirelessly reports to a decision-making edge server. At the same time, the transmissions are operated with Finite Blocklength (FBL) codes to achieve low-latency transmission. We introduce the task of Age upon Decision (AuD) aimed at the timeliness of tasks used for decision-making, which highlights the timeliness of the information at decision-making moments. For the case in which dynamic task generation and random fading channels are considered, we provide a task AuD minimization design by jointly selecting UE and allocating blocklength. In particular, to solve the task AuD minimization problem, we transform the optimization problem to a Markov Decision Process problem and propose an Error Probability-Controlled Action-Masked Proximal Policy Optimization (EMPPO) algorithm. Via simulation, we show that the proposed design achieves a lower AuD than baseline methods across various network conditions, especially in scenarios with significant channel Signal-to-Noise Ratio (SNR) differences and low average SNR, which shows the robustness of EMPPO and its potential for real-time applications. [ABSTRACT FROM AUTHOR]

Our increasingly connected world continues to face an ever-growing number of network-based attacks. An Intrusion Detection System (IDS) is an essential security technology used for detecting these attacks. Although numerous Machine Learning-based IDSs have been proposed for the detection of malicious network traffic, the majority have difficulty properly detecting and classifying the more uncommon attack types. In this paper, we implement a novel hybrid technique using synthetic data produced by a Generative Adversarial Network (GAN) to use as input for training a Deep Reinforcement Learning (DRL) model. Our GAN model is trained on the NSL-KDD dataset, a publicly available collection of labeled network traffic data specifically designed to support the evaluation and benchmarking of IDSs. Ultimately, our findings demonstrate that training the DRL model on synthetic datasets generated by specific GAN models can result in better performance in correctly classifying minority classes over training on the true imbalanced dataset. [ABSTRACT FROM AUTHOR]

The efficient planning of electric power systems is essential to meet both the current and future energy demands. In this context, reinforcement learning (RL) has emerged as a promising tool for control problems modeled as Markov decision processes (MDPs). Recently, its application has been extended to the planning and operation of power systems. This study provides a systematic review of advances in the application of RL and deep reinforcement learning (DRL) in this field. The problems are classified into two main categories: Operation planning including optimal power flow (OPF), economic dispatch (ED), and unit commitment (UC) and expansion planning, focusing on transmission network expansion planning (TNEP) and distribution network expansion planning (DNEP). The theoretical foundations of RL and DRL are explored, followed by a detailed analysis of their implementation in each planning area. This includes the identification of learning algorithms, function approximators, action policies, agent types, performance metrics, reward functions, and pertinent case studies. Our review reveals that RL and DRL algorithms outperform conventional methods, especially in terms of efficiency in computational time. These results highlight the transformative potential of RL and DRL in addressing complex challenges within power systems. [ABSTRACT FROM AUTHOR]

Assessments of the hosting capacity of electricity distribution networks are of paramount importance, as they facilitate the seamless integration of rooftop photovoltaic systems into the grid, accelerating the transition towards a more carbon neutral and sustainable system. This paper employs a deep reinforcement learning-based approach to evaluate the real-time hosting capacity of low voltage distribution networks in a model-free manner. The proposed approach only requires real-time customer voltage data and solar irradiation data to provide a fast and accurate estimate of real-time hosting capacity at each customer connection point. This study addresses the imperative for accurate electrical models, which are frequently unavailable, in evaluating the hosting capacity of electricity distribution networks. To meet this challenge, the proposed approach utilizes a deep neural network-based, data-driven model of a low-voltage electricity distribution network. This proposed methodology incorporates model-free elements, enhancing its adaptability and robustness. In addition, a comparative analysis between model-based and model-free hosting capacity assessment methods is presented, highlighting their respective strengths and weaknesses. The utilization of the proposed hosting capacity estimation model enables distribution network service providers to make well-informed decisions regarding grid planning, leading to cost minimization. [ABSTRACT FROM AUTHOR]

Label accuracy plays an important role in supervised hyperspectral image classification problems. However, labelling can be prone to various types of noise, mainly including human errors and incorrect pseudo-label predictions. Learning from datasets with noisy labels is a critical challenge in practice. To address this issue, we first propose a deep reinforcement learning method named HyperRefine to tackle the label noise problem for hyperspectral image classification. The method utilizes an agent to learn the accuracy of each sample label and determine its suitability to be used in training the classifier model. The agent is trained using a reinforcement signal based on the reward obtained from a small validation set that reflects performance on the target task, where the validation set is generated from the training set by a momentum-based recruiter. Compared to complex hand-crafted methods of label noise cleaning, HyperRefine is easy to implement, yet highly effective. The robustness of the model to different types of label noises is significantly improved by further combining a novel label refinement approach. Experiments on artificially added label noise and pseudo-label noise hyperspectral datasets show that the proposed method can achieve promising performance. [ABSTRACT FROM AUTHOR]