Search

Deep reinforcement learning is an artificial intelligence technology that combines deep learning and reinforcement learning and has been widely applied in multiple fields. As a type of deep reinforcement learning algorithm, the A3C (Asynchronous Advantage Actor-Critic) algorithm can effectively utilize computer resources and improve training efficiency by synchronously training Actor-Critic in multiple threads. Inspired by the excellent performance of the A3C algorithm, this paper uses the A3C algorithm to solve the UUV (Unmanned Underwater Vehicle) collision avoidance planning problem in unknown environments. This collision avoidance planning algorithm can have the ability to plan in real-time while ensuring a shorter path length, and the output action space can meet the kinematic constraints of UUVs. In response to the problem of UUV collision avoidance planning, this paper designs the state space, action space, and reward function. The simulation results show that the A3C collision avoidance planning algorithm can guide a UUV to avoid obstacles and reach the preset target point. The path planned by this algorithm meets the heading constraints of the UUV, and the planning time is short, which can meet the requirements of real-time planning. [ABSTRACT FROM AUTHOR]

Ongoing research endeavors are exploring the potential of artificial intelligence to enhance the efficiency of wireless communication systems. Nevertheless, complex computational mechanisms, such as those inherent in neural networks, are not optimally suited for applications where the reduction of computational intricacy is of paramount importance. The rise in Bluetooth-enabled devices has led to the widespread adoption of Bluetooth Low Energy (BLE) in various IoT applications, primarily due to its low power consumption. For specific applications, such as lost and found tags which operate on small batteries, it's especially important to further reduce power usage. With the objective of achieving low power consumption by optimally selecting channels and advertisement intervals, this paper introduces a parameter selection method derived from the Multi-Armed Bandit (MAB) algorithm, a technique known for addressing human decision-making challenges. In this study, we evaluate our proposed method using simulations in diverse environments. The outcomes indicate that, without compromising much on reliability, our approach can reduce power consumption by up to 40% based on the wireless surroundings. Additionally, when this method was implemented on an actual BLE device, it demonstrated effectiveness in reducing power consumption by about 35% in real environments. [ABSTRACT FROM AUTHOR]

Weapon target assignment (WTA) is a typical problem in the command and control of modern warfare. Despite the significance of the problem, traditional algorithms still have shortcomings in terms of efficiency, solution quality, and generalization. This paper presents a novel multi-objective evolutionary optimization algorithm (MOEA) that integrates a deep Q-network (DQN)-based adaptive mutation operator and a greedy-based crossover operator, designed to enhance the solution quality for the multi-objective WTA (MO-WTA). Our approach (NSGA-DRL) evolves NSGA-II by embedding these operators to strike a balance between exploration and exploitation. The DQN-based adaptive mutation operator is developed for predicting high-quality solutions, thereby improving the exploration process and maintaining diversity within the population. In parallel, the greedy-based crossover operator employs domain knowledge to minimize ineffective searches, focusing on exploitation and expediting convergence. Ablation studies revealed that our proposed operators significantly boost the algorithm performance. In particular, the DQN mutation operator shows its predictive effectiveness in identifying candidate solutions. The proposed NSGA-DRL outperforms state-and-art MOEAs in solving MO-WTA problems by generating high-quality solutions. [ABSTRACT FROM AUTHOR]

Facility Management is a multi-disciplinary task in which coordination is key to attaining success during the building life cycle and for which technology assumes an increasing role. This sector is demanding more available and accurate tools to optimize the management process, decrease the probability of failure, and reduce the time spent on anomaly analysis. So, the present paper presents work developed to improve access to building anomaly recognition and to predict the building degradation state in an automatized way. The methodology applied to achieve this goal started with a survey and digital data acquisition from a case study, followed by the automatized detection of building anomalies using supervised classification in Deep Learning; then, the early diagnosis of threatening conditions for building degradation took place using degradation curves based on data records and regression algorithms. The results drive this study a step forward toward obtaining advanced tools for Facility Management based in Artificial Intelligence, able to provide the most appropriate moment at which to intervene according to the cost-benefit. The present work provided better results on the harmonic mean of precision and recall when compared with previous studies of image classification for the construction sector. Moreover, the mathematical functions for the prediction of future degradation based on the data field for each construction system were presented and can be applied to the typologies of other buildings. In the end, future developments and limitations are highlighted. [ABSTRACT FROM AUTHOR]

Recently, higher structure complicacy and performance requirements of the aero-engine have brought higher demands on its control system. With the rapid development of artificial intelligence technology, the intelligent controller with self-learning ability will be able to make a great difference. In the paper, we propose an aero-engine intelligent controller design method based on twin delayed deep deterministic policy gradient (TD3) algorithm. The design method allows the intelligent controller to interact autonomously with the aero-engine system to acquire the optimal control sequence. The JT9D turbofan engine is used to introduce the controller design workflow proposed in the paper. First, the problem of aero-engine control is described as a Markov decision process for deep reinforcement learning (DRL) algorithms. Second, a complete intelligent controller design process is constructed by reasonably designing the network structures and reward function. Finally, the comparison simulations are carried out to verify the superior performance of the controller design methods. The simulation results indicate that low-pressure turbine speed has no overshoot, and the settling time does not exceed 0.88s during the engine acceleration process. In the deceleration process, the overshoot of the low-pressure turbine speed is limited to 0.74% and the settling time does not exceed about 0.6s. The results prove that the TD3 controller outperforms deep deterministic policy gradient (DDPG) and the proportional-integral-derivative (PID) in the speed tracking control. [ABSTRACT FROM AUTHOR]

There has been a lot of research on edge-computing task offloading in deep reinforcement learning (DRL). Deep reinforcement learning is one of the important algorithms in the current AI field, but there is still room for improvement in the time cost and adaptive correction ability of the algorithm. This paper studies the application of DRL algorithms in edge-computing task offloading, and its key innovation is to propose an MADRLCO algorithm, which is based on the design idea of the Actor–Critic framework, uses the DNN model to act as an Actor, and can more accurately locate the initial decision through iterative training, and use the LSTM model to optimize the Critic, which can be more accurate. The optimal decision can be located in a short period of time. The main work of this paper is divided into three parts: (1) The AC algorithm of the Actor–Critic framework in DRL is proposed to be applied to edge-computing task offloading. (2) To address the weak generalization ability of the basic version of the Actor–Critic algorithm in multi-objective optimization, the sequential quantitative correction and adaptive correction parameter K method are used to optimize the Critic frame, thereby improving the generalization ability of the model in multi-objective decision-making and improving the rationality of decision-making results. (3) Aiming at the problem of large time cost in the critical framework of the model, a search algorithm for resource allocation-related parameters based on the time-series prediction method is proposed (time-series forecasting is a research branch of pattern recognition), which reduces the time overhead of the algorithm and improves the adaptive correction capability of the model. The algorithm in this paper can adapt to not only the time-varying network channel state, but also the time-varying number of device connections. Finally, it is proved by experiments that compared with the DRL calculation offloading algorithm based on DNN plus binary search, the MADRLCO algorithm reduces the model training time by 66.27%, and in the environment of the time-varying number of devices in the metasystem, the average model average. The standard calculation rate is 0.0403 higher than that of the current optimal algorithm. [ABSTRACT FROM AUTHOR]

Traditional path planning is mainly utilized for path planning in discrete action space, which results in incomplete ship navigation power propulsion strategies during the path search process. Moreover, reinforcement learning experiences low success rates due to its unbalanced sample collection and unreasonable design of reward function. In this paper, an environment framework is designed, which is constructed using the Box2D physics engine and employs a reward function, with the distance between the agent and arrival point as the main, and the potential field superimposed by boundary control, obstacles, and arrival point as the supplement. We also employ the state-of-the-art PPO (Proximal Policy Optimization) algorithm as a baseline for global path planning to address the issue of incomplete ship navigation power propulsion strategy. Additionally, a Beta policy-based distributed sample collection PPO algorithm is proposed to overcome the problem of unbalanced sample collection in path planning by dividing sub-regions to achieve distributed sample collection. The experimental results show the following: (1) The distributed sample collection training policy exhibits stronger robustness in the PPO algorithm; (2) The introduced Beta policy for action sampling results in a higher path planning success rate and reward accumulation than the Gaussian policy at the same training time; (3) When planning a path of the same length, the proposed Beta policy-based distributed sample collection PPO algorithm generates a smoother path than traditional path planning algorithms, such as A*, IDA*, and Dijkstra. [ABSTRACT FROM AUTHOR]

Field of robotics has been under the limelight because of recent advances in Artificial Intelligence (AI). Due to increased diversity in multi-agent systems, new models are being developed to handle complexity of such systems. However, most of these models do not address problems such as; uncertainty handling, efficient learning, agent coordination and fault detection. This paper presents a novel approach of implementing Reinforcement Learning (RL) on hierarchical robotic search teams. The proposed algorithm handles uncertainties in the system by implementing Q-learning and depicts enhanced efficiency as well as better time consumption compared to prior models. The reason for that is each agent can take action on its own thus there is less dependency on leader agent for RL policy. The performance of this algorithm is measured by introducing agents in an unknown environment with both Markov Decision Process (MDP) and RL policies at their disposal. Simulation-based comparison of the agent motion is presented using the results from of MDP and RL policies. Furthermore, qualitative comparison of the proposed model with prior models is also presented. [ABSTRACT FROM AUTHOR]

The rapid technological development of computing power and system operations today allows for increasingly advanced algorithm implementation, as well as path planning in real time. The objective of this article is to provide a structured review of simulations and practical implementations of collision-avoidance and path-planning algorithms in autonomous underwater vehicles (AUVs). The novelty of the review paper is to consider not only the results of numerical research but also the newest results of verifying collision-avoidance and path-planning algorithms in real applications together with a comparison of the difficulties encountered during simulations and their practical implementation. Analysing the last 20 years of AUV development, it can be seen that experiments in a real environment are dominated by classical methods. In the case of simulation studies, artificial intelligence (AI) methods are used as often as classical methods. In simulation studies, the APF approach is most often used among classical methods, whereas among AI algorithms reinforcement learning and fuzzy logic methods are used. For real applications, the most used approach is reactive behaviors, and AI algorithms are rarely used in real implementations. Finally, this article provides a general summary, future works, and a discussion of the limitations that inhibit the further development in this field. [ABSTRACT FROM AUTHOR]

Hybrid electric vehicles are, nowadays, considered as one of the most promising technologies for reducing on-road greenhouse gases and pollutant emissions. Such a goal can be accomplished by developing an intelligent energy management system which could lead the powertrain to exploit its maximum energetic performances under real-world driving conditions. According to the latest research in the field of control algorithms for hybrid electric vehicles, Reinforcement Learning has emerged between several Artificial Intelligence approaches as it has proved to retain the capability of producing near-optimal solutions to the control problem even in real-time conditions. Nevertheless, an accurate design of both agent and environment is needed for this class of algorithms. Within this paper, a detailed plan for the complete project and development of an energy management system based on Q-learning for hybrid powertrains is discussed. An integrated modular software framework for co-simulation has been developed and it is thoroughly described. Finally, results have been presented about a massive testing of the agent aimed at assessing for the change in its performance when different training parameters are considered. [ABSTRACT FROM AUTHOR]

In this paper, we propose a novel adaptive importance sampling (AIS) algorithm for probabilistic inference. The sampler learns a proposal distribution adaptation strategy by framing AIS as a reinforcement learning problem. Under this structure, the proposal distribution of the sampler is treated as an agent whose state is controlled using a parameterized policy. At each iteration of the algorithm, the agent earns a reward that is related to its contribution to the variance of the AIS estimator of the normalization constant of the target distribution. Policy gradient methods are employed to learn a locally optimal policy that maximizes the expected value of the sum of all rewards. Numerical simulations on two different examples demonstrate promising results for the future application of the proposed method to complex Bayesian models. [ABSTRACT FROM AUTHOR]

• A multi-constraint model for impulsive orbital pursuit-evasion game (OPEG) is constructed. • A learning-based algorithm (i.e. PRD-MADDPG) for impulsive OPEG is designed. • The Predict-Reward-Detect (PRD) training framework for improving MADDPG is established. • The PRD-MADDPG performs better than conventional MADDPG in impulsive OPEGs. • The winning mechanism of the game for both players under multiple factors is analyzed. This paper comprehensively investigates the problem of impulsive orbital pursuit-evasion games (OPEGs) by using an artificial intelligence-based approach. First, the mathematical model for the impulsive OPEGs in which the pursuer and evader both perform their orbital maneuvers by imposing the impulsive velocity increments is constructed. Second, the problem of impulsive OPEGs is transformed into a bilateral optimization problem with a minimum–maximum optimization index in terms of terminal time and multiple constraints such as maneuverability, total fuel consumption, and mission time, etc. To determine the optimal impulsive maneuvers for both sides, a PRD-MADDPG (Predict-Reward-Detect Multi-Agent Deep Deterministic Policy Gradient) algorithm in the frame of multi-agent reinforcement learning is designed. This novel algorithm uses the basic MADDPG to achieve the strategies training and learning, and applies the supplemental PRD to predict the change of game state during the interval between two adjacent impulsive maneuvers and incorporate these information into the algorithm training in the form of predicted reward. Finally, some pursuit-evasion missions near the Geosynchronous Earth Orbit are numerically analyzed to verify the validness and effectiveness of the algorithm. The results prove that the PRD-MADDPG algorithm is very efficient to find applicable strategies even considering rather complex constraints. It is also shown that the learning-based strategies can be effectively applied in the extended scenarios which are not seen in the training process. [ABSTRACT FROM AUTHOR]

Autonomous vehicles or self-driving cars are prevalent nowadays, many vehicle manufacturers, and other tech companies are trying to develop autonomous vehicles. One major goal of the self-driving algorithms is to perform manoeuvres safely, even when some anomaly arises. To solve these kinds of complex issues, Artificial Intelligence and Machine Learning methods are used. One of these motion planning problems is when the tires lose their grip on the road, an autonomous vehicle should handle this situation. Thus the paper provides an Autonomous Drifting algorithm using Reinforcement Learning. The algorithm is based on a model-free learning algorithm, Twin Delayed Deep Deterministic Policy Gradients (TD3). The model is trained on six different tracks in a simulator, which is developed specifically for autonomous driving systems; namely CARLA. [ABSTRACT FROM AUTHOR]

This paper addresses the challenge of minimizing training time for the control of Heating, Ventilation, and Air-conditioning (HVAC) systems with online Reinforcement Learning (RL). This is done by developing a novel approach to Multi-Agent Reinforcement Learning (MARL) to HVAC systems. In this paper, the environment formed by the HVAC system is formulated as a Markov Game (MG) in a general sum setting. The MARL algorithm is designed in a decentralized structure, where only relevant states are shared between agents, and actions are shared in a sequence, which are sensible from a system's point of view. The simulation environment is a domestic house located in Denmark and designed to resemble an average house. The heat source in the house is an air-to-water heat pump, and the HVAC system is an Underfloor Heating system (UFH). The house is subjected to weather changes from a data set collected in Copenhagen in 2006, spanning the entire year except for June, July, and August, where heat is not required. It is shown that: (1) When comparing Single Agent Reinforcement Learning (SARL) and MARL, training time can be reduced by 70% for a four temperature-zone UFH system, (2) the agent can learn and generalize over seasons, (3) the cost of heating can be reduced by 19% or the equivalent to 750 kWh of electric energy per year for an average Danish domestic house compared to a traditional control method, and (4) oscillations in the room temperature can be reduced by 40% when comparing the RL control methods with a traditional control method. [ABSTRACT FROM AUTHOR]

Grid-interactive building control poses a critical challenge in the context of grid modernization and decarbonization. Recently, various artificial intelligence-based optimal control approaches have been proposed, providing innovative solutions to optimal building control problems. However, researchers and practitioners are now confronted with a dilemma when selecting appropriate control strategies, ranging from model-based (knowledge-incorporating) to learning-based (data-driven) approaches, and hybrid methods combining them. Although each algorithm in existing literature claims superiority over specific baselines, their performance has never been systematically compared and analyzed, owing to the absence of a unified platform for comprehensive evaluation. To fill this knowledge gap, we identify and implement all state-of-the-art approaches within a modular training and evaluation framework, assessing their efficacy in a grid-interactive building control problem. In this paper, we also introduce a streamlined hybrid method that complements existing hybrid approaches. Our comparative study reveals and quantifies the advantages of hybrid methods: on average, they achieve near-optimal control while requiring less than 14% of the online computation of traditional model-based methods. To achieve this performance, they need as few as 2% of the training samples of purely learning-based methods. Finally, we provide insights into the merits, limitations, and implementation of each method to help researchers better understand the state of the art and future directions. The software framework implemented in this study is open-sourced to facilitate future research. [ABSTRACT FROM AUTHOR]

Machine learning technologies are being integrated into robotic systems faster to enhance their efficacy and adaptability in dynamic environments. The primary goal of this research was to propose a method to develop an Autonomous Mobile Robot (AMR) that integrates Simultaneous Localization and Mapping (SLAM), odometry, and artificial vision based on deep learning (DL). All are executed on a high-performance Jetson Nano embedded system, specifically emphasizing SLAM-based obstacle avoidance and path planning using the Adaptive Monte Carlo Localization (AMCL) algorithm. Two Convolutional Neural Networks (CNNs) were selected due to their proven effectiveness in image and pattern recognition tasks. The ResNet18 and YOLOv3 algorithms facilitate scene perception, enabling the robot to interpret its environment effectively. Both algorithms were implemented for real-time object detection, identifying and classifying objects within the robot's environment. These algorithms were selected to evaluate their performance metrics, which are critical for real-time applications. A comparative analysis of the proposed DL models focused on enhancing vision systems for autonomous mobile robots. Several simulations and real-world trials were conducted to evaluate the performance and adaptability of these models in navigating complex environments. The proposed vision system with CNN ResNet18 achieved an average accuracy of 98.5%, a precision of 96.91%, a recall of 97%, and an F1-score of 98.5%. However, the YOLOv3 model achieved an average accuracy of 96%, a precision of 96.2%, a recall of 96%, and an F1-score of 95.99%. These results underscore the effectiveness of the proposed intelligent algorithms, robust embedded hardware, and sensors in robotic applications. This study proves that advanced DL algorithms work well in robots and could be used in many fields, such as transportation and assembly. As a consequence of the findings, intelligent systems could be implemented more widely in the operation and development of AMRs. [ABSTRACT FROM AUTHOR]

Deep reinforcement learning (DRL) is a promising technology to address the resource allocation problem for efficient data transmission in complex network environments. However, most DRL-based resource allocation algorithms suffer from limited feature extraction capabilities and lack scalability and generalization, especially in heterogeneous Industrial Internet of Things (IIoT) environments. In this paper, we develop a lifelong learning architecture that can integrate artificial intelligence (AI) algorithms into the heterogeneous IIoT network for efficient data transmission. Based on this, we propose an intelligent resource allocation algorithm based on dual-attention DRL (DADR) for forwarding node selection and channel access slot allocation in a specific network environment. The proposed DADR algorithm combines the advantages of multi-dimension convolutional attention and multi-head self-attention mechanisms. It can provide local- and global-feature fusion capabilities for distributed nodes while maximizing the performance of data transmission. Furthermore, we present a lifelong federated meta reinforcement learning (LFMRL) that can effectively utilize prior knowledge and enable the DRL agent quickly adapt to a new environment. Specifically, LFMRL adopts a federated meta learning-based knowledge fusion algorithm to fuse the knowledge of learned DADR algorithms and iteratively update the shared model, thereby improving the scalability and generalization of the shared model in heterogeneous IIoT environments. In addition, a simple and efficient knowledge transfer mechanism is enabled to accelerate the DRL model convergence by transferring the knowledge of the shared model to the new environment. Simulation results demonstrate the effectiveness of the proposed algorithms in terms of energy efficiency, data transmission reliability, and network stability. Compared to DADR and FedAvg algorithms, LFMRL algorithm can further reduce the energy consumption, training time, and average forwarding node switching times, while improving packet delivery rate to 99.2%. • A novel lifelong learning architecture for resource allocation in heterogeneous IIoT. • Dual-attention deep reinforcement learning provides local- and global- feature fusion. • Federated meta learning based knowledge fusion learns the commonness of models. • Reinforcement learning based knowledge transfer learns the properties of model. • Knowledge fusion and knowledge transfer provide lifelong learning ability for system. [ABSTRACT FROM AUTHOR]

Hydrogen provides a clean source of energy that can be produced with the aid of electrolysers. For electrolysers to operate cost-effectively and safely, it is necessary to define an appropriate maintenance strategy. Predictive maintenance is one of such strategies but often relies on data from sensors which can also become faulty, resulting in false information. Consequently, maintenance will not be performed at the right time and failure will occur. To address this problem, the artificial intelligence concept is applied to make predictions on sensor readings based on data obtained from another instrument within the process. In this study, a novel algorithm is developed using Deep Reinforcement Learning (DRL) to select the best feature(s) among measured data of the electrolyser, which can best predict the target sensor data for predictive maintenance. The features are used as input into a type of deep neural network called long short-term memory (LSTM) to make predictions. The DLR developed has been compared with those found in literatures within the scope of this study. The results have been excellent and, in fact, have produced the best scores. Specifically, its correlation coefficient with the target variable was practically total (0.99). Likewise, the root-mean-square error (RMSE) between the experimental sensor data and the predicted variable was only 0.1351. [ABSTRACT FROM AUTHOR]

The close‐range autonomous air combat has gained significant attention from researchers involved in applications related to artificial intelligence (AI). A majority of the previous studies on autonomous air combat were focused on one‐on‐one air combat scenarios, however, the modern air combat is mostly conducted in formations. With regard to the aforementioned factors, a novel hierarchical maneuvering control architecture is introduced that is applied to the multi‐aircraft close‐range air combat scenario, which can handle air combat scenarios with variable‐size formation. Subsequently, three air combat sub‐tasks are designed, and recurrent soft actor‐critic (RSAC) algorithm combined with competitive self‐play (SP) is incorporated to learn the sub‐strategies. A novel hierarchical multi‐agent reinforcement learning (HMARL) algorithm is proposed to obtain the high‐level strategy for target and sub‐strategy selection. The training performance of the training algorithm of sub‐strategies and high‐level strategy in different air combat scenarios is evaluated. The obtained strategies are analyzed and it is found that the formations exhibit effective cooperative behavior in symmetric and asymmetric scenarios. Finally, the ideas of engineering implementation of the maneuvering control architecture are given. The study provides a solution for future multi‐aircraft autonomous air combat. [ABSTRACT FROM AUTHOR]

In the current world of the Internet of Things, cyberspace, mobile devices, businesses, social media platforms, healthcare systems, etc., there is a lot of data online today. Machine learning (ML) is something we need to understand to do smart analyses of these data and make smart, automated applications that use them. There are many different kinds of machine learning algorithms. The most well-known ones are supervised, unsupervised, semi-supervised, and reinforcement learning. This article goes over all the different kinds of machine-learning problems and the machine-learning algorithms that are used to solve them. The main thing this study adds is a better understanding of the theory behind many machine learning methods and how they can be used in the real world, such as in energy, healthcare, finance, autonomous driving, e-commerce, and many more fields. This article is meant to be a go-to resource for academic researchers, data scientists, and machine learning engineers when it comes to making decisions about a wide range of data and methods to start extracting information from the data and figuring out what kind of machine learning algorithm will work best for their problem and what results they can expect. Additionally, this article presents the major challenges in building machine learning models and explores the research gaps in this area. In this article, we also provided a brief overview of data protection laws and their provisions in different countries. [ABSTRACT FROM AUTHOR]

Aiming at the poor robustness and adaptability of traditional control methods for different situations, the deep deterministic policy gradient (DDPG) algorithm is improved by designing a hybrid function that includes different rewards superimposed on each other. In addition, the experience replay mechanism of DDPG is also improved by combining priority sampling and uniform sampling to accelerate the DDPG's convergence. Finally, it is verified in the simulation environment that the improved DDPG algorithm can achieve accurate control of the robot arm motion. The experimental results show that the improved DDPG algorithm can converge in a shorter time, and the average success rate in the robotic arm end-reaching task is as high as 91.27%. Compared with the original DDPG algorithm, it has more robust environmental adaptability. [ABSTRACT FROM AUTHOR]

Reformulating the history matching problem from a least-square mathematical optimization problem into a Markov Decision Process introduces a method in which reinforcement learning can be utilized to solve the problem. This method provides a mechanism where an artificial deep neural network agent can interact with the reservoir simulator and find multiple different solutions to the problem. Such a formulation allows for solving the problem in parallel by launching multiple concurrent environments enabling the agent to learn simultaneously from all the environments at once, achieving significant speed up. [ABSTRACT FROM AUTHOR]

The pH treatment unit is widely used in various processes, such as wastewater treatment, pharmaceutical manufacturing, and fermentation. It is essential to get the on-specifications product. Thus, controlling pH is key management for accomplishing the manufacturing objective. However, the highly nonlinear pH characteristics of acid–base titration make pH regulation difficult. Applications of artificial intelligence for process control have progressed and gained popularity recently. The development of reinforcement learning (RL) control with a deep deterministic policy gradient (DDPG) algorithm to handle coupled pH and liquid level control in a continuous stirred tank reactor with a strong acid–base reaction is presented in this study. To validate the RL model, the reward functions are created individually for the level and pH controls. The grid search technique is deployed to optimize the hyperparameters of the RL controller models, including the number of nodes in the hidden layers and the number of episodes. The control performance of the proposed RL control system was compared with that of the proportional-integral controller in a servo-regulatory test. The simulation results show that the proposed RL controllers outperform the proportional-integral controllers in approaching setpoints faster, with better performance and less oscillation. [ABSTRACT FROM AUTHOR]

Artificial intelligence for aircraft guidance is a hot research topic, and deep reinforcement learning is one of the promising methods. However, due to the different movement patterns of destinations in different guidance tasks, it is inefficient to train agents from scratch. In this article, a policy-reuse algorithm based on destination position prediction is proposed to solve this problem. First, the reward function is optimized to improve flight trajectory quality and training efficiency. Then, by predicting the possible termination position of the destinations in different moving patterns, the problem is transformed into a fixed-position destination aircraft guidance problem. Last, taking the agent in the fixed-position destination scenario as the baseline agent, a new guidance agent can be trained efficiently. Simulation results show that this method can significantly improve the training efficiency of agents in new tasks, and its performance is stable in tasks with different similarities. This research broadens the application scope of the policy-reuse approach and also enlightens the research in other fields. [ABSTRACT FROM AUTHOR]

With the development of artificial intelligence technology, the behavior decision-making of an intelligent smart marine autonomous surface ship (SMASS) has become particularly important. This research proposed local path planning and a behavior decision-making approach based on improved Proximal Policy Optimization (PPO), which could drive an unmanned SMASS to the target without requiring any human experiences. In addition, a generalized advantage estimation was added to the loss function of the PPO algorithm, which allowed baselines in PPO algorithms to be self-adjusted. At first, the SMASS was modeled with the Nomoto model in a simulation waterway. Then, distances, obstacles, and prohibited areas were regularized as rewards or punishments, which were used to judge the performance and manipulation decisions of the vessel Subsequently, improved PPO was introduced to learn the action–reward model, and the neural network model after training was used to manipulate the SMASS's movement. To achieve higher reward values, the SMASS could find an appropriate path or navigation strategy by itself. After a sufficient number of rounds of training, a convincing path and manipulation strategies would likely be produced. Compared with the proposed approach of the existing methods, this approach is more effective in self-learning and continuous optimization and thus closer to human manipulation. [ABSTRACT FROM AUTHOR]

Deep Reinforcement Learning (DRL), one of the most popular research topics in artificial intelligence, has achieved a breakthrough in continuous control tasks. Nonetheless, the DRL algorithm's instability and local optimality have a bad influence impact on its performance. The Deep Deterministic Policy Gradients (DDPG) algorithm uses a "soft" update to slow down the target value rate of change to alleviate this problem. However, there is still a specific target approximate error variance. The variance will aggravate the degree of the data dispersion and reduce the stability of the model. This paper proposed the DDPG with averaged state-action estimation (Averaged-DDPG) algorithm. It aims to minimize the adverse effects of conflict, which calculates the action reward by averaging the estimated values of previously learned Q values, thus reducing the training process's fluctuation and improving the algorithm's performance. The evaluation results in continuous control tasks show that Averaged-DDPG can enhance the agent's learning efficiency and training balance more effectively than the original DDPG algorithm. [ABSTRACT FROM AUTHOR]

The Deep Reinforcement Learning (DRL) algorithm, Proximal Policy Optimization (PPO2), is deployed on a custom spacecraft (S/C) build and loss model to determine if an Artificial Intelligence (AI) can learn to monitor satellite constellation health and determine an optimal replacement strategy. A custom environment is created to simulate how S/C are built, launched, generate revenue, and finally decay. The reinforcement learning agent successfully learned an optimal policy for two models: a Simplified Model where the financial cost of actions is ignored; and an Advanced Model where the financial cost of actions is a major element. In both models the AI monitors the constellations and takes multiple strategic and tactical actions to replace satellites to maintain constellation performance. The Simplified Model showed that the PPO2 algorithm was able to converge on an optimal solution after ∼ 200,000 simulations. The Advanced Model was much more difficult for the AI to learn, and thus, the performance drops during the early episodes, but eventually converges to an optimal policy at ∼ 25,000,000 simulations. With the Advanced Model, the AI is taking actions that are successfully providing strategies for constellation management and satellite replacements which include these actions' financial implications. Thus, the methods in this paper provide initial research developments towards a real-world tool and an AI application that can aid various Aerospace businesses in managing Low Earth Orbit (LEO) constellations. This type of AI application may become imperative for deploying and maintaining small satellite mega-constellations. [ABSTRACT FROM AUTHOR]

Artificial intelligence (AI) techniques in power grid control and energy management in building automation require both deep Q-networks (DQNs) and deep deterministic policy gradients (DDPGs) in deep reinforcement learning (DRL) as off-policy algorithms. Most studies on improving the stability of DRL have addressed these with replay buffers and a target network using a delayed temporal difference (TD) backup, which is known for minimizing a loss function at every iteration. The loss functions were developed for DQN and DDPG, and it is well-known that there have been few studies on improving the techniques of the loss functions used in both DQN and DDPG. Therefore, we modified the loss function based on a temporal consistency (TC) loss and adapted the proposed TC loss function for the target network update in both DQN and DDPG. The proposed TC loss function showed effective results, particularly in a critic network in DDPG. In this work, we demonstrate that, in OpenAI Gym, both "cart-pole" and "pendulum", the proposed TC loss function shows enormously improved convergence speed and performance, particularly in the critic network in DDPG. [ABSTRACT FROM AUTHOR]