Search

The control of the entrainment and diffusion of the heat and mass ejected along the jet flow requires the control of fluid motion. In the present study, optimal initial velocity distributions for suppressing and promoting jet entrainment and diffusion were identified using deep reinforcement learning under constant initial jet flow rate conditions. Two-dimensional jet flow simulations using OpenFOAM were combined with a deep Q-network as the learning algorithm. The spatial distribution of the initial jet (velocity and ejection angle) changed. The results show that to suppress the entrainment of the jet, an optimal initial velocity distribution shape in which the velocity in the centre is almost 0, with an inward velocity just outside of it, and a large outward velocity at the outer edge, is desirable. However, to promote entrainment, a shape with a large velocity in the centre portion and small velocity at the outer edge, with an overall inward angle, is desirable. To suppress the diffusion of the jet, an optimal initial velocity distribution shape with a large velocity in the centre portion, a small velocity at the outer edge, and an overall inward angle is desirable. Conversely, to promote diffusion, a shape with a large outward velocity at both the centre and outer edges is desirable. It is also indicated that, when the initial velocity distribution of the jet has a special shape, and the relationship between flow rate, centre velocity, and centre temperature that is normally observed in a jet flow may not hold. In other words, by changing the initial velocity distribution, the entrainment and diffusion of the jet can be separated and controlled, which is usually difficult to achieve using conventional engineering methods., Version of Record 12 October 2023

With the emergence of cyber-physical systems (CPSs) in utility systems like electricity, water, and gas networks, data collection has become more prevalent. While data collection in these systems has numerous advantages, it also raises concerns about privacy as it can potentially reveal sensitive information about users. To address this issue, we propose a Bayesian approach to control the adversarial inference and mitigate the physical-layer privacy problem in CPSs. Specifically, we develop a control strategy for the worst-case scenario where an adversary has perfect knowledge of the user’s control strategy. For finite state-space problems, we derive the fixed-point Bellman’s equation for an optimal stationary strategy and discuss a few practical approaches to solve it using optimization-based control design. Addressing the computational complexity, we propose a reinforcement learning approach based on the Actor-Critic architecture. To also support smart meter privacy research, we present a publicly accessible “Co-LivEn” dataset with comprehensive electrical measurements of appliances in a co-living household. Using this dataset, we benchmark the proposed reinforcement learning approach. The results demonstrate its effectiveness in reducing privacy leakage. Our work provides valuable insights and practical solutions for managing adversarial inference in cyber-physical systems, with a particular focus on enhancing privacy in smart meter applications., QC 20240226

The building sector is the single largest contributor to greenhouse gas emissions, making it a natural focal point for reducing energy consumption. More efficient use of energy is also becoming increasingly important for property managers as global energy prices are skyrocketing. This report is conducted on behalf of Sustainable Intelligence, a Swedish company that specializes in building automation solutions. It investigates whether deep reinforcement learning (DLR) algorithms can be implemented in a building control environment, if it can be more effective than traditional solutions, and if it can be achieved in reasonable time. The algorithms that were tested were Deep Deterministic Policy Gradient, DDPG, and Proximal Policy Optimization, PPO. They were implemented in a simulated BOPTEST environment in Brussels, Belgium, along with a traditional heating curve and a PI-controller for benchmarks. DDPG never converged, but PPO managed to reduce energy consumption compared to the best benchmark, while only having slightly worse thermal discomfort. The results indicate that DRL algorithms can be implemented in a building environment and reduce green house gas emissions in a reasonable training time. This might especially be interesting in a complex building where DRL can adapt and scale better than traditional solutions. Further research along with implementations on physical buildings need to be done in order to determine if DRL is the superior option.

Autonomous navigation is essential to remotely operating mobile vehicles on Mars, as communication takes up to 20 minutes to travel between the Earth and Mars. Several autonomous navigation methods have been implemented in Mars rovers and other mobile robots, such as odometry or simultaneous localization and mapping (SLAM) until the past few years when deep reinforcement learning (DRL) emerged as a viable alternative. In this thesis, a simulation model for end-to-end DRL Mars rover autonomous navigation training was created using Unity Engine, using local inputs such as GNSS, LiDAR, and gyro. This model was then trained in navigation in a flat environment using the proximal policy optimization (PPO) algorithm. The results of the training and future work are discussed.

Communication is an effective mechanism for coordinating the behaviors of multiple agents, broadening their views of the environment, and to support their collaborations. In the field of multi-agent deep reinforcement learning (MADRL), agents can improve the overall learning performance and achieve their objectives by communication. Agents can communicate various types of messages, either to all agents or to specific agent groups, or conditioned on specific constraints. With the growing body of research work in MADRL with communication (Comm-MADRL), there is a lack of a systematic and structural approach to distinguish and classify existing Comm-MADRL approaches. In this paper, we survey recent works in the Comm-MADRL field and consider various aspects of communication that can play a role in designing and developing multi-agent reinforcement learning systems. With these aspects in mind, we propose 9 dimensions along which Comm-MADRL approaches can be analyzed, developed, and compared. By projecting existing works into the multi-dimensional space, we discover interesting trends. We also propose some novel directions for designing future Comm-MADRL systems through exploring possible combinations of the dimensions.

The computational multiphase flow community has grappled with mass loss in the level set method for decades. Numerous solutions have been proposed, from fixing the reinitialization step to combining the level set method with other conservative schemes. However, our work reveals a more fundamental culprit: the smooth Heaviside and delta functions inherent to the standard formulation. We propose a novel approach using variational analysis to incorporate a mass conservation constraint. It introduces a Lagrange multiplier that enforces overall mass balance. Notably, as the delta function sharpens, i.e., approaches the Dirac delta limit, the Lagrange multiplier approaches zero. However, the exact Lagrange multiplier method disrupts the signed distance property of the level set function. This motivates us to develop an approximate version of the Lagrange multiplier that preserves both overall mass and signed distance property of the level set function. Our framework even recovers existing mass-conserving level set methods, revealing some inconsistencies in prior analyses. We extend this approach to three-phase flows for fluid-structure interaction (FSI) simulations. Rigorous test problems confirm that the FSI dynamics produced by our simple, easy-to-implement immersed formulation with the approximate Lagrange multiplier method are accurate and match state-of-the-art solvers.Next, we develop a simulation infrastructure to perform fully resolved simulations of wave energy converter (WEC) devices. We use the fictitious domain Brinkman penalization (FD/BP) technique, which is computationally more efficient than the body-conforming grid techniques. Simulating WEC devices involves complex fluid-structure interactions and, if done accurately, can be used to test different types of controllers in a more realistic setting. We simulate the dynamics of an inertial sea wave energy converter (ISWEC) device and a heaving vertical cylinder point absorber device in a numerical wave tank

Traditional production scheduling and resource allocation methods often struggle to adapt to changing conditions in manufacturing environments. To address this challenge, this study leverages an adaptive control system integrated with a Deep Deterministic Policy Gradient (DDPG) alongside a particle swarm optimization algorithm to enable real-time production scheduling and allocation of resources. The system continuously learns from generated production data and adjusts production schedules with resource allocations based on evolving conditions such as demand fluctuations and resource availability. By harnessing the capabilities of Deep Reinforcement learning, the proposed approach of applying the DDPG algorithm to simulate the environment improves production efficiency, minimizes delays, and optimizes resource utilization. Through conducted experiments, the effectiveness of the DDPG-Particle Swarm Optimization technique (DRPO) was demonstrated in enhancing dynamic production scheduling and resource allocation in simulated manufacturing settings. This study presents a significant step towards intelligent, self-improving production control systems that can navigate complex and dynamic manufacturing environments.

Water contamination in extensive aquatic resources is a pressing issue, especially during current drought conditions across the world. To adress this, a novel approach involving a heterogeneous sensing capabilities fleet of four autonomous surface vehicles is introduced for efficient contamination mapping. To reduce costs, vehicles may be equipped with low quality sensors meaning measurements reliability differs between vehicles and affects model accuracy. The diverse sensing capabilities are characterized by a wide range of sensor standard deviations, addressing the applicability of the framework in real-world scenarios with commercial sensors. This research leverages Gaussian Processes to accurately model spatial distribution of contamination, integrating measurements from the vehicles to understand contamination patterns comprehensively. Additionally, an informative path planning strategy is introduced based on a centralized neural network which implements a Double Deep Q-Learning algorithm, driving the decision-making process of all agents. Effective learning hinges on accurately defining the observation and reward functions, for which several proposals will be compared. These tailored definitions are essential for guiding the learning process, and minimizing the error towards the main goal: to obtain the best possible contamination model. Remarkably, the proposed system demonstrates superior performance in Ypacaraí Lake scenario, surpassing traditional heuristics like lawn mower or particle swarm optimization by up to 82% in reducing mean squared error in highly contaminated regions for several combinations of agents.

Actor-critic deep reinforcement learning methods have demonstrated significant performance in many challenging decision-making and control tasks, but also suffer from high sample complexity and overestimation bias. Current researches focus on using underestimation to balance overestimation and reducing bias through ensemble learning, but introducing underestimation bias and excessive network costs. In this paper, we first analyze the effect of action selection policy on estimation bias. Then, we propose the Explorer-Actor-Critic (EAC) method that gives a more conservative objective for the actor to reduce overestimation, introduces a learnable explorer to improve exploration ability, and uses an action mixing mechanism to mitigate experience distribution bias. Furthermore, we apply the EAC method to TD3 and SAC and verify its effectiveness through extensive comparison and ablation experiments. Our algorithm not only outperforms state-of-the-art algorithms, but also is compatible with other actor-critic methods.

As an essential resource management problem in network virtualization, virtual network embedding (VNE) aims to allocate the finite resources of physical network to sequentially arriving virtual network requests (VNRs) with different resource demands. Since this is an NP-hard combinatorial optimization problem, many efforts have been made to provide viable solutions. However, most existing approaches have either ignored the admission control of VNRs, which has a potential impact on long-term performances, or not fully exploited the temporal and topological features of the physical network and VNRs. In this article, we propose a deep Hierarchical Reinforcement Learning approach to learn a joint Admission Control and Resource Allocation policy for VNE, named HRL-ACRA. Specifically, the whole VNE process is decomposed into an upper-level policy for deciding whether to admit the arriving VNR or not and a lower-level policy for allocating resources of the physical network to meet the requirement of VNR through the HRL approach. Considering the proximal policy optimization as the basic training algorithm, we also adopt the average reward method to address the infinite horizon problem of the upper-level agent and design a customized multi-objective intrinsic reward to alleviate the sparse reward issue of the lower-level agent. Moreover, we develop a deep feature-aware graph neural network to capture the features of VNR and physical network and exploit a sequence-to-sequence model to generate embedding actions iteratively. Finally, extensive experiments are conducted in various settings, and show that HRL-ACRA outperforms state-of-the-art baselines in terms of both the acceptance ratio and long-term average revenue.

High-accuracy GNSS positioning in urban environments is important for applications like safe autonomous driving, however, dynamic errors in complex urban environments limit positioning performances. Recently, deep learning-based (DL) approaches can obtain better GNSS positioning solutions in complex urban environments than model-based ones. However, DL-based approaches simply concentrate one-view GNSS observations as inputs, which are insufficient to model vehicle states accurately, and temporally continuous observations are highly correlated, leading to inaccurate positioning correction results. To solve the challenge, we propose a Sparse Representation-based Multiview Deep Reinforcement Learning model for positioning correction, which employs attention-based multiview fusion to process multiview observations, and uses sparse representation to alleviate disturbances from highly correlated observations. To represent the vehicle state sufficiently, we build a multiview positioning correction environment, and develop an attention-weighted multiview fusion module to fuse temporal features as belief states based on adaptively learned attention weights. To effectively process redundant and correlated multiview features, we impose the ℓ1 norm regularizer to learn sparse hidden representations and improve the precision of value estimation. Finally, we construct a sparse representation-driven multiview actor-critic positioning correction model to achieve high-accuracy GNSS positioning in complex urban environments. We validate performances in both Google Smartphone Decimeter Challenge (GSDC) datasets and our collected GNSS datasets in the Guangzhou area (GZGNSS). Experimental results show that our algorithm can improve localization performances with 27% improvements from WLS+KF in GSDC trajectories, 16% from RTK, and 6% from DL-based methods in GZGNSS trajectories. © The Author(s), under exclusive licence to Springer-Verlag GmbH Germany, part of Springer Nature 2024.

Recently, artificial intelligence paves the way for the development of smart services for people anytime and anywhere, which poses great challenges on accessing computing resources. Multi-Access edge computing complements existing cloud computing infrastructure at the edge of the network, where mobile users can offload computationally intensive tasks of smart applications to edge servers that are in proximity to the users themselves. Existing offloading schemes mainly focus on selecting edge servers for each offloading task with the goal of optimizing the overall average latency. However, the solutions with the optimal overall average latency may be not the most suitable for all offloading tasks. There is still a possibility that offloading leads to an extreme case of ultra-high latency, which is not acceptable for latency-sensitive applications. To address this problem, we therefore introduce modern portfolio theory (MPT) to jointly consider the overall average latency and potential risks in optimal edge server selection. The task offloading problem is regarded as an investment portfolio with the objective of maximizing the 'return' while minimizing the risk. Combining MPT with deep reinforcement learning (DRL), we design two proximal policy optimization (PPO)-based task offloading algorithms to jointly optimize these two objectives. The algorithm computes a portfolio for each mobile user that enables the diversification of edge server selection, thereby minimizing the risk and the average latency. Extensive simulation results based on three real-world trace datasets show that our algorithms significantly outperform the state-of-The-Art solutions and can reduce the overall average latency and the risk by 59% and 85% at most, respectively. © 2002-2012 IEEE.

Edge caching is a promising solution for next-generation networks by empowering caching units in small-cell base stations (SBSs), which allows user equipments (UEs) to fetch users’ requested contents that have been pre-cached in SBSs. It is crucial for SBSs to predict accurate popular contents through learning while protecting users’ personal information. Traditional federated learning (FL) can protect users’ privacy but the data discrepancies among UEs can lead to a degradation in model quality. Therefore, it is necessary to train personalized local models for each UE to predict popular contents accurately. In addition, the cached contents can be shared among adjacent SBSs in next-generation networks, thus caching predicted popular contents in different SBSs may affect the cost to fetch contents. Hence, it is critical to determine where the popular contents are cached cooperatively. To address these issues, we propose a cooperative edge caching scheme based on elastic federated and multi-agent deep reinforcement learning (CEFMR) to optimize the cost in the network. We first propose an elastic FL algorithm to train the personalized model for each UE, where adversarial autoencoder (AAE) model is adopted for training to improve the prediction accuracy, then a popular content prediction algorithm is proposed to predict the popular contents for each SBS based on the trained AAE model. Finally, we propose a multi-agent deep reinforcement learning (MADRL) based algorithm to decide where the predicted popular contents are collaboratively cached among SBSs. Our experimental results demonstrate the superiority of our proposed scheme to existing baseline caching schemes. IEEE

This research investigates the use of Deep Reinforcement Learning (DRL) to solve a multi-objective optimization problem (MOOP) in manufacturing. The research focuses on a simplified real-world scenario involving five ovens. The goal is to optimize both the utilization and capacity rates of the ovens by improving the sequence and batch of products processed. Different processing times and oven capacities complicate real-time decision-making for product dispatching, specifically the sequence and combination of products for each oven at each dispatch time. This increases the challenge of efficiency in DRL policy learning. Existing literature mainly uses evolutionary algorithms to solve MOOP, with few studies using DRL for optimizing Batching and Sequencing Problems (BSP), such as those for ovens. In this study, we conducted experiments using standard Proximal Policy Optimization (PPO) and Deep Q-Network (DQN) models from a prior project and proposed a hybrid model that combines a standard PPO model with a parameter transfer strategy. This hybrid approach aims to enhance exploration and policy learning efficiency. The objective, defined as the average effective utilization rate (utilization rate multiplied by capacity rate), is decomposed into multiple subproblems using a weighted sum function. The neural network parameters are trained on one subproblem and transferred to neighboring subproblems, enabling the agent to explore from a good starting point for each subproblem. A simulation model serves as the environment for interacting with the DRL models. Based on 100 individual tests on unseen data for the hybrid model and comparison models (random rule, heuristic rule, and PPO and DQN models from a prior project), the hybrid model produces Pareto optimal solutions with a higher average effective utilization rate. Additionally, it reduces training time to 1/24th of that required by the previous PPO and DQN models.

Optimizing renewable energy usage in smart microgrids that contain photovoltaic production and battery storage is important due to the potential to reduce overall CO2 emissions and thus lead to more environmental friendly energy systems. However, due to the complex nature of energy management in smart grids and the volatile nature of energy production from PV systems, the problem is complex to solve. In this work we aim to optimize the energy in a microgrid comprising six houses using a digital twin based approach based on Deep Reinforcement Learning. We develop a Soft Actor-Critic (SAC) agent to address this intricate challenge, with the aim to simultaneously reduce emissions, maintain user comfort, while maximizing grid efficiency and resiliency to cope with spurious grid outages. We propose and evaluate different reward functions that guide the agent in finding its optimal policy. Furthermore, we discuss the implications of our results and outline potential future steps, envisioning ongoing refinement and advancements in our pursuit of optimal solutions for the complex interplay of severaal objectives in microgrid management., AI4ENERGY

This study aims to investigate the efficacy of Deep Reinforcement Learning (DRL) in the domain of industrial motion control, especially path planning, and scheduling. In order to abstract challenges that can be found in industrial path planning and scheduling operations, we construct a simulation of a generic logistical use case for an AGV. The experiments are centered on evaluating the performance of PPO, benchmarked against a deterministic agent in controlling an AGV for optimal scheduling and routing decisions. The agents’ control relates to finding the optimal scheduling strategy of machines, while efficiently navigating in a complex operational framework under risk of disruptions and time-out events. We compare the characteristics of deterministic algorithms to DRL, contrast the results of the experiment, and derive implications for the industries, Masterarbeit Universität Innsbruck 2024

The recent advances in space technology are focusing on fully autonomous, real-time, long-term orbit management and mission planning for large-scale satellite constellations in Low-Earth Orbit (LEO). Thus, a pioneering approach for autonomous orbital station-keeping has been introduced using a model-free Deep Policy Gradient-based Reinforcement Learning (DPGRL) strategy explicitly tailored for LEO. Addressing the critical need for more efficient and self-regulating orbit management in LEO satellite constellations, this work explores the potential synergy between Deep Reinforcement Learning (DRL) and Neuro-Evolution of Augmenting Topology (NEAT) to optimize station-keeping strategies with the primary goal to empower satellite to autonomously maintain their orbit in the presence of external perturbations within an allowable tolerance margin, thereby significantly reducing operational costs while maintaining precise and consistent station-keeping throughout their life cycle. The study specifically tailors DPGRL algorithms for LEO satellites, considering low-thrust constraints for maneuvers and integrating dense reward schemes and domain-based reward shaping techniques. By showcasing the adaptability and scalability of the combined NEAT and DRL framework in diverse operational scenarios, this approach holds immense promise for revolutionizing autonomous orbit management, paving the way for more efficient and adaptable satellite operations while incorporating the physical constraints of satellite, such as thruster limitations., Funder: OPTACOM (OPC-OSE-CC-0536);Full text license: CC BY

Deep reinforcement learning (DRL) is a promising solution for voltage control of distribution grids with high penetration of inverter-based renewable energy sources (RESs). Yet, when adopting the DRL-based control method, the safe and optimal operation of the system cannot be guaranteed at the same time, as the conventional DRL agent is not designed to solve the hard constraint problem. To address this challenge, this paper proposes a deep neural network (DNN) assisted projection based DRL method for safe control of distribution grids. First, a finite iteration projection algorithm is proposed to guarantee hard constraints by converting a non-convex optimization problem into a finite iteration problem. Next, a DNN assisted projection method is proposed to accelerate the calculation of projection and achieve the practical implementation of hard constraints in DRL problem. Finally, a DNN Projection embedded twin-delayed deep deterministic policy gradient (DPe-TD3) method is proposed to achieve optimal operation of distribution grids with guaranteed 100% safety of the distribution grid. The safety of the DRL training is guaranteed via the embedded Projection DNN in TD3 with participation in gradient return process, which could smoothly and effectively project the DRL agent actions into the feasible area, thus guaranteeing the safety of data driven control and the optimal operation at the same time. The case studies and comparisons are conducted in the IEEE 33 bus system to show the effectiveness of the proposed method., QC 20240709

To obtain a suitable scheduling scheme in an effective time range, the minimum completion time is taken as the objective of Flexible Job Shop scheduling Problems (FJSP) with different scales, and Composite Dispatching Rules (CDRs) are applied to generate feasible solutions. Firstly, the binary tree coding method is adopted, and the constructed function set is normalized. Secondly, a CDR mining approach based on an Improved Genetic Programming Algorithm (IGPA) is designed. Two population initialization methods are introduced to enrich the initial population, and a superior and inferior population separation strategy is designed to improve the global search ability of the algorithm. At the same time, two individual mutation methods are introduced to improve the algorithm’s local search ability, to achieve the balance between global search and local search. In addition, the effectiveness of the IGPA and the superiority of CDRs are verified through comparative analysis. Finally, Deep Reinforcement Learning (DRL) is employed to solve the FJSP by incorporating the CDRs as the action set, the selection times are counted to further verify the superiority of CDRs., QC 20240527

The wide integration of inverter based renewable energy sources (RESs) in modern grids may cause severe voltage violation issues due to high stochastic fluctuations of RESs. Existing centralized approaches can achieve optimal results for voltage regulation, but they have high communication burdens; existing decentralized methods only require local information, but they cannot achieve optimal results. Deep reinforcement learning (DRL) based methods are effective to deal with uncertainties, but it is difficult to guarantee secure constraints in existing DRL training. To address the above challenges, this paper proposes a projection embedded multi-agent DRL algorithm to achieve decentralized optimal control of distribution grids with guaranteed 100% safety. The safety of the DRL training is guaranteed via an embedded safe policy projection, which could smoothly and effectively restrict the DRL agent action space, and avoid any violation of physical constraints in distribution grid operations. The multi-agent implementation of the proposed algorithm enables the optimal solution achieved in a decentralized manner that does not require real-time communication for practical deployment. The proposed method is tested in modified IEEE 33-bus distribution and compared with existing methods; the results validate the effectiveness of the proposed method in achieving decentralized optimal control with guaranteed 100% safety and without the requirement of real-time communications., QC 20240429

The integration of human–robot collaboration yields substantial benefits, particularly in terms of enhancing flexibility and efficiency within a range of mass-personalized manufacturing tasks, for example, small-batch customized product inspection and assembly/disassembly. Meanwhile, as human–robot collaboration lands broader in manufacturing, the unstructured scene and operator uncertainties are increasingly involved and considered. Consequently, it becomes imperative for robots to execute in a safe and adaptive manner rather than solely relying on pre-programmed instructions. To tackle it, a systematic solution for safe robot motion generation in human–robot collaborative activities is proposed, leveraging mixed-reality technologies and Deep Reinforcement Learning. This solution covers the entire process of collaboration starting with an intuitive interface that facilitates bare-hand task command transmission and scene coordinate transformation before the collaboration begins. In particular, mixed-reality devices are employed as effective tools for representing the state of humans, robots, and scenes. This enables the learning of an end-to-end Deep Reinforcement Learning policy that addresses both the uncertainties in robot perception and decision-making in an integrated manner. The proposed solution also implements policy simulation-to-reality deployment, along with motion preview and collision detection mechanisms, to ensure safe robot motion execution. It is hoped that this work could inspire further research in human–robot collaboration to unleash and exploit the powerful capabilities of mixed reality., QC 20240418

Unmanned aerial vehicles (UAVs) are becoming increasingly valuable as a new type of mobile communication device and autonomous decision-making device in many application areas, including the Internet of Things (IoT). UAVs have advantages over other stationary devices in terms of high flexibility. However, a UAV, as a mobile device, still faces some challenges in optimizing its trajectory for data collection. Firstly, the high complexity of the movement action and state space of the UAV’s 3D trajectory is not negligible. Secondly, in unknown urban environments, a UAV must avoid obstacles accurately in order to ensure a safe flight. Furthermore, without a priori wireless channel characterization and ground device locations, a UAV must reliably and safely complete the data collection from the ground devices under the threat of unknown interference. All of these require the proposing of intelligent and automatic onboard trajectory optimization techniques. This paper transforms the trajectory optimization problem into a Markov decision process (MDP), and deep reinforcement learning (DRL) is applied to the data collection scenario. Specifically, the double deep Q-network (DDQN) algorithm is designed to address intelligent UAV trajectory planning that enables energy-efficient and safe data collection. Compared with the traditional algorithm, the DDQN algorithm is much better than the traditional Q-Learning algorithm, and the training time of the network is shorter than that of the deep Q-network (DQN) algorithm., QC 20240516

This thesis explores the application of Reinforcement Learning (RL) to production job scheduling, specifically within the context of a factory environment. We developed an environment reflecting the dynamic operations of a factory floor, incorporating machines, jobs, and operations to be performed on jobs called recipes. We train Deep Q-Network (DQN) and Advantage Actor Critic (A2C) RL agents with function approximation using neural networks (NN) since the state space is continuous. An important aspect of our work is the development of an RL agent capable of directly assigning jobs to machines without relying on dispatch rules. Instead, the actions taken by the agent are job-machine pairings. Throughout the training process, the environment, observation space, and reward structures were incrementally refined based on agent performance. The primary objective is to minimize the average number of jobs completed late, termed as tardiness. Our results demonstrate that RL agents can learn to make effective scheduling decisions, showing promising improvements in minimizing total jobs completed late compared to the traditional dispatch rules.

5G has emerged to revolutionize the telecommunications industry. With its many possibilities, there are also great challenges, such as maintaining the increased complexity of themany parameters in these new networks. It is a common practice to test new features of thenetworks before employing them, and this is often done in a simulated environment. Thetask of this thesis was to investigate if self-adaptive software, in simulations at Ericsson,could dynamically change the bandwidth to increase the net throughput while minimizingthe packet loss, i.e. to maximize the overall quality of service on the network, without theneed of human intervention. A simple simulation of a 5G network was created to trainand test the effect of two proposed AI-models. The models tested were Proximal PolicyOptimization and Deep Deterministic Policy Gradient, where the former model showedpromising results while the latter did not yield any significant improvements comparedto the benchmarks. The study indicates that self-adaptive software, in simulated environments, can effectively be achieved by using AI while increasing the quality of service.

Multi-agent path planning is applied in a wide range of applications in robotics and autonomous vehicles, including aerial vehicles such as drones and other unmanned aerial vehicles (UAVs), to solve tasks in areas like surveillance, search and rescue, and transportation. In today's rapidly evolving technology in the fields of automation and artificial intelligence, multi-agent path planning is growing increasingly more relevant. The main problems encountered in multi-agent path planning are collision avoidance with other agents, obstacle evasion, and pathfinding from a starting point to an endpoint. In this project, the objectives were to create intelligent agents capable of navigating through two-dimensional eight-agent cost map environments to a static target, while avoiding collisions with other agents and simultaneously minimizing the path cost. The method of reinforcement learning was used by utilizing the development platform Unity and the open-source ML-Agents toolkit that enables the development of intelligent agents with reinforcement learning inside Unity. Perlin Noise was used to generate the cost maps. The reinforcement learning algorithm Proximal Policy Optimization was used to train the agents. The training was structured as a curriculum with two lessons, the first lesson was designed to teach the agents to reach the target, without colliding with other agents or moving out of bounds. The second lesson was designed to teach the agents to minimize the path cost. The project successfully achieved its objectives, which could be determined from visual inspection and by comparing the final model with a baseline model. The baseline model was trained only to reach the target while avoiding collisions, without minimizing the path cost. A comparison of the models showed that the final model outperformed the baseline model, reaching an average of $27.6\%$ lower path cost., Multi-agent-vägsökning används inom en rad olika tillämpningar inom robotik och autonoma fordon, inklusive flygfarkoster såsom drönare och andra obemannade flygfarkoster (UAV), för att lösa uppgifter inom områden som övervakning, sök- och räddningsinsatser samt transport. I dagens snabbt utvecklande teknik inom automation och artificiell intelligens blir multi-agent-vägsökning allt mer relevant. De huvudsakliga problemen som stöts på inom multi-agent-vägsökning är kollisioner med andra agenter, undvikande av hinder och vägsökning från en startpunkt till en slutpunkt. I detta projekt var målen att skapa intelligenta agenter som kan navigera genom tvådimensionella åtta-agents kostnadskartmiljöer till ett statiskt mål, samtidigt som de undviker kollisioner med andra agenter och minimerar vägkostnaden. Metoden förstärkningsinlärning användes genom att utnyttja utvecklingsplattformen Unity och Unitys open-source ML-Agents toolkit, som möjliggör utveckling av intelligenta agenter med förstärkningsinlärning inuti Unity. Perlin Brus användes för att generera kostnadskartorna. Förstärkningsinlärningsalgoritmen Proximal Policy Optimization användes för att träna agenterna. Träningen strukturerades som en läroplan med två lektioner, den första lektionen var utformad för att lära agenterna att nå målet, utan att kollidera med andra agenter eller röra sig utanför gränserna. Den andra lektionen var utformad för att lära agenterna att minimera vägkostnaden. Projektet uppnådde framgångsrikt sina mål, vilket kunde fastställas genom visuell inspektion och genom att jämföra den slutliga modellen med en basmodell. Basmodellen tränades endast för att nå målet och undvika kollisioner, utan att minimera vägen kostnaden. En jämförelse av modellerna visade att den slutliga modellen överträffade baslinjemodellen, och uppnådde en genomsnittlig $27,6\%$ lägre vägkostnad.

Detection of abnormalities within the inner ear is a challenging task even for experienced clinicians. In this study, we propose an automated method for automatic abnormality detection to provide support for the diagnosis and clinical management of various otological disorders. We propose a framework for inner ear abnormality detection based on deep reinforcement learning for landmark detection which is trained uniquely in normative data. In our approach, we derive two abnormality measurements: Dimage and Uimage. The first measurement, Dimage, is based on the variability of the predicted configuration of a well-defined set of landmarks in a subspace formed by the point distribution model of the location of those landmarks in normative data. We create this subspace using Procrustes shape alignment and Principal Component Analysis projection. The second measurement, Uimage, represents the degree of hesitation of the agents when approaching the final location of the landmarks and is based on the distribution of the predicted Q-values of the model for the last ten states. Finally, we unify these measurements in a combined anomaly measurement called Cimage. We compare our method's performance with a 3D convolutional autoencoder technique for abnormality detection using the patch-based mean squared error between the original and the generated image as a basis for classifying abnormal versus normal anatomies. We compare both approaches and show that our method, based on deep reinforcement learning, shows better detection performance for abnormal anatomies on both an artificial and a real clinical CT dataset of various inner ear malformations with an increase of 11.2% of the area under the ROC curve. Our method also shows more robustness against the heterogeneous quality of the images in our dataset.

Conventional Air Traffic Control is still predominantly being done by human Air Traffic Controllers, however, as the traffic density increases, the workload of the controllers increases as well. Especially for the area of unmanned aviation, driven by the rise in drones, having human controllers might become unfeasible. One of the methods that is currently being investigated for replacing the conflict resolution task of Air Traffic Control is Reinforcement Learning. As violation of the required separation margins, also called an intrusion, is an event of relatively low frequency, using Reinforcement Learning for this task comes with difficulties that can potentially be attributed to data imbalance. This paper artificially increased the traffic density during the training phase of the Reinforcement Learning method to investigate what the importance is of a balanced data set on the performance of the Reinforcement Learning method. It was found that as the traffic density increased, the Reinforcement Learning methods started to outperform the analytical methods. Beyond this it was found that methods trained at higher traffic densities, but tested at lower traffic densities, outperformed the methods trained at that specific density. This indicates that it might be better to always ensure that the training scenarios are more complex than anticipated during the execution phase, even if that results in unrealistic scenarios., Control & Simulation

Virtualized Radio Access Networks (vRANs) are fully configurable and can be implemented at a low cost over commodity platforms to enable network management flexibility. In this paper, a novel vRAN reconfiguration problem is formulated to jointly reconfigure the functional splits of the base stations (BSs), locations of the virtualized central units (vCUs) and distributed units (vDUs), their resources, and the routing for each BS data flow. The objective is to minimize the long-term total network operation cost while adapting to the varying traffic demands and resource availability. In the first step, testbed measurements are performed to study the relationship between the traffic demands and computing resources, which reveals high variance and depends on the platform and its load. Consequently, finding the perfect model of the underlying system is non-trivial. Therefore, to solve the proposed problem, a deep reinforcement learning (RL)-based framework is proposed and developed using model-free RL approaches. Moreover, the problem consists of multiple BSs sharing the same resources, which results in a multi-dimensional discrete action space and leads to a combinatorial number of possible actions. To overcome this curse of dimensionality, action branching architecture, which is an action decomposition method with a shared decision module followed by neural network is combined with Dueling Double Deep Q-network (D3QN) algorithm. Simulations are carried out using an O-RAN compliant model and real traces of the testbed. Our numerical results show that the proposed framework successfully learns the optimal policy that adaptively selects the vRAN configurations, where its learning convergence can be further expedited through transfer learning even in different vRAN systems. It also offers significant cost savings by up to 59% of a static benchmark, 35% of Deep Deterministic Policy Gradient with discretization, and 76% of non-branching D3QN., Networked Systems

The volume of instant delivery has witnessed a significant growth in recent years. Given the involvement of numerous heterogeneous stakeholders, instant delivery operations are inherently characterized by dynamics and uncertainties. This study introduces two order dispatching strategies, namely task buffering and dynamic batching, as potential solutions to address these challenges. The task buffering strategy aims to optimize the assignment timing of orders to couriers, thereby mitigating demand uncertainties. On the other hand, the dynamic batching strategy focuses on alleviating delivery pressure by assigning orders to couriers based on their residual capacity and extra delivery distances. To model the instant delivery problem and evaluate the performances of order dispatching strategies, Adaptive Agent-Based Order Dispatching (ABOD) approach is developed, which combines agent-based modelling, deep reinforcement learning, and the Kuhn-Munkres algorithm. The ABOD effectively captures the system's uncertainties and heterogeneity, facilitating stakeholders learning in novel scenarios and enabling adaptive task buffering and dynamic batching decision-makings. The efficacy of the ABOD approach is verified through both synthetic and real-world case studies. Experimental results demonstrate that implementing the ABOD approach can lead to a significant increase in customer satisfaction, up to 275.42%, while simultaneously reducing the delivery distance by 11.38% compared to baseline policies. Additionally, the ABOD approach exhibits the ability to adaptively adjust buffering times to maintain high levels of customer satisfaction across various demand scenarios. As a result, this approach offers valuable support to logistics providers in making informed decisions regarding order dispatching in instant delivery operations., Transport and Planning

The deployment of beyond 5G and 6G network infrastructures will enable highly dynamic services requiring stringent Quality of Service (QoS). Supporting such combinations in today’s transport networks will require high flexibility and automation to operate near real-time and reduce overprovisioning. Many solutions for autonomous network operation based on Machine Learning require a global network view, and thus need to be deployed at the Software-Defined Networking (SDN) controller. In consequence, these solutions require implementing control loops, where algorithms running in the controller use telemetry measurements collected at the data plane to make decisions that need to be applied at the data plane. Such control loops fit well for provisioning and failure management purposes, but not for near real-time operation because of their long response times. In this paper, we propose a distributed approach for autonomous near-real-time flow routing with QoS assurance. Our solution brings intelligence closer to the data plane to reduce response times; it is based on the combined application of Deep Reinforcement Learning (DRL) and Multi-Agent Systems (MAS) to create a distributed collaborative network control plane. Node agents ensure QoS of traffic flows, specifically end-to-end delay, while minimizing routing costs by making distributed routing decisions. Algorithms in the centralized network controller provide the agents with the set of routes that can be used for each traffic flow and give freedom to the agents to use them during operation. Results show that the proposed solution is able to ensure end-to-end delay under the desired maximum and greatly reduce routing costs. This performance is achieved in dynamic scenarios without previous knowledge of the traffic profile or the background traffic, for single domain and multidomain networks., The research leading to these results has received funding from the European Commission thought the HORIZON SNS JU DESIRE6G (G.A. 101096466), the MINECO UNICO5G TIMING (TSI-063000-2021-145) and the MICINN IBON (PID2020-114135RB-I00) projects, and from the ICREA Institution., Peer Reviewed, Postprint (author's final draft)

1. The United Nations has declared 2021–2030 the decade on ecosystem restoration with the aim of preventing, stopping and reversing the degradation of the ecosystems of the world, often caused by the fragmentation of natural landscapes. Human activities separate and surround habitats, making them too small to sustain viable animal populations or too far apart to enable foraging and gene flow. Despite the need for strategies to solve fragmentation, it remains unclear how to efficiently reconnect nature. In this paper, we illustrate the potential of deep reinforcement learning (DRL) to tackle the spatial optimisation aspect of connectivity conservation planning.2. The propensity of spatial optimisation problems to explode in complexity depending on the number of input variables and their states is and will continue to be one of its most serious obstacles. DRL is an emerging class of methods focused on training deep neural networks to solve decision-making tasks and has been used to learn good heuristics for complex optimisation problems. While the potential of DRL to optimise conservation decisions seems huge, only few examples of its application exist.3. We applied DRL to two real-world raster datasets in a connectivity planning setting, targeting graph-based connectivity indices for optimisation. We show that DRL converges to the known optimums in a small example where the objective is the overall improvement of the Integral Index of Connectivity and the only constraint is the budget. We also show that DRL approximates high-quality solutions on a large example with additional cost and spatial configuration constraints where the more complex Probability of Connectivity Index is targeted. To the best of our knowledge, there is no software that can target this index for optimisation on raster data of this size.4. DRL can be used to approximate good solutions in complex spatial optimisation problems even when the conservation feature is non-linear like graph-based indic

In the Condition-Based Maintenance (CBM) context, the definition of optimal maintenance plans for an aircraft fleet depends on an efficient integration of : (i) the probabilistic predictions of the health condition of the components and (ii) the stochastic arrival of the corrective maintenance tasks, together with consideration of the preventive maintenance tasks as defined in the Maintenance Planning Document (MPD). To this end, in this paper, we present a two-stage dynamic scheduling framework to solve the aircraft fleet maintenance scheduling problem under a CBM strategy in a disruptive environment. In the first stage of the framework, we address the uncertainty in the predicted health state of the monitored components by planning the optimal maintenance policy based upon the belief state-space of the health of the components. The decision-making process is formulated as a Partially Observable Markov Decision Process (POMDP) and is solved using the Partially Observable Monte Carlo Planning (POMCP) algorithm, considering the aircraft maintenance scheduling problem requirements. In the second stage, a Deep Q-Network (DQN) is developed, that integrates the defined maintenance policy of the monitored components within the scheduling of the aircraft fleet's preventive and corrective maintenance tasks. Our model, through a rolling horizon approach, continuously creates and adjusts the maintenance schedule, reacting to new updated task information, where the availability of maintenance resources constraints the execution of each task. The proposed framework was tested on a case study from a large airline and the performance was evaluated against the current state practice of the airline. The results show that our model can schedule 96.4% of monitored components on-time. As a consequence of this, a 46.2% maintenance cost reduction is achieved for the considered monitored components relative to a corrective maintenance approach., Air Transport & Operations

This paper considers the problem of real-time dispatching of a fleet of automated guided vehicles (AGVs) with battery constraints. AGVs must be immediately assigned to transport requests, which arrive randomly. In addition, the AGVs must be repositioned and recharged, awaiting future transport requests. Each transport request has a soft time window with late delivery incurring a tardiness cost. This research aims to minimize the total costs, consisting of tardiness costs of transport requests and travel costs of AGVs. We extend the existing literature by making a distinction between parking and charging nodes, where AGVs wait idle for incoming transporting requests and satisfy their charging needs, respectively. Also, we formulate this online decision-making problem as a Markov decision process and propose a solution approach based on deep reinforcement learning. To assess the quality of the proposed approach, we compare it with the optimal solution of a mixed-integer linear programming model that assumes full knowledge of transport requests in hindsight and hence serves as a lower-bound on the costs. We also compare our solution with a heuristic policy used in practice. We assess the performance of the proposed solutions in an industry case study using real-world data.

Moving to a multi-cloud environment and service-based architecture, 5G and future 6G networks require additional defensive mechanisms to protect virtualized network resources. This paper presents MERLINS, a novel architecture generating optimal Moving Target Defense (MTD) policies for proactive and reactive security of network slices. By formally modeling telecommunication networks compliant with Network Function Virtualization (NFV) into a multi-objective Markov Decision Process (MOMDP), MERLINS uses deep Reinforcement Learning (deep-RL) to optimize the MTD strategy that considers security, network performance, and service level requirements. Practical experiments on a 5G testbed showcase the feasibility as well as restrictions of MTD operations and the effectiveness in mitigating malware infections. It is observed that multi-objective RL (MORL) algorithms outperform state-of-the-art deep-RL algorithms that scalarize the reward vector of the MOMDP. This improvement by a factor of two leads to a better MTD policy than the baseline static counterpart used for the evaluation.

This paper investigates the application of deep reinforcement learning (DRL) algorithm for the decision-support of a prosumer aggregation in a hierarchical local electricity market (LEM) comprising a peer-to-peer (P2P) market and a corrective market. The agent first submits bids/asks to the P2P market where prosumer aggregations are able to trade electricity directly with each other. After that, the agent participates in the corrective market, where the market operator formulates the corrective market as an AC optimal power flow (OPF) problem to ensure the system is operated within its operational limits. A DRL algorithm, namely Twin Delayed Deep Deterministic Policy Gradient (TD3), is used to find the strategic bidding strategy. The algorithm is tested on a real medium-voltage distribution grid to evaluate the effectiveness of the strategic bidding method. The result of the case study demonstrates that the agent can derive trading strategies to obtain high profits based on the TD3 algorithm.

The One-Warehouse Multi-Retailer (OWMR) system is the prototypical distribution and inventory system. Many OWMR variants exist, e.g. demand in excess of supply may be completely back-ordered, partially back-ordered, or lost. Prior research has focused on the study of heuristic reordering policies such as echelon base-stock levels coupled with heuristic allocation policies. Constructing well-performing policies is time-consuming and must be redone for every problem variant. By contrast, Deep Reinforcement Learning (DRL) is a general purpose technique for sequential decision making that has yielded good results for various challenging inventory systems. However, applying DRL to OWMR problems is nontrivial, since allocation involves setting a quantity for each retailer: The number of possible allocations grows exponentially in the number of retailers. Since each action is typically associated with a neural network output node, this renders standard DRL techniques intractable. Our proposed DRL algorithm instead inferences a multi-discrete action distribution which has output nodes that grow linearly in the number of retailers. Moreover, when total retailer orders exceed the available warehouse inventory, we propose a random rationing policy that substantially improves the ability of standard DRL algorithms to train good policies because it promotes the learning of feasible retailer order quantities. The resulting algorithm outperforms general-purpose benchmark policies by ∼1−3% for the lost sales case and by ∼12−20% for the partial back-ordering case. For complete back-ordering, the algorithm cannot consistently outperform the benchmark.

We introduce a novel heuristic designed to address the supply chain inventory management problem in the context of a two-echelon divergent supply chain. The proposed heuristic advances the current state-of-the-art by combining deep reinforcement learning with multi-stage stochastic programming. In particular, deep reinforcement learning is employed to determine the number of batches to produce, while multi-stage stochastic programming is applied to make shipping decisions. To support further research, we release a publicly available software environment that simulates a wide range of two-echelon divergent supply chain settings, allowing the manipulation of various parameter values, including those associated with seasonal demands. We then present a comprehensive set of numerical experiments considering constraints on production and warehouse capacities under fixed and variable logistic costs. The results demonstrate that the proposed heuristic significantly and consistently outperforms pure deep reinforcement learning algorithms in minimizing total costs. Moreover, it overcomes several inherent limitations of multi-stage stochastic programming models, thus underscoring its potential advantages in addressing complex supply chain scenarios.

This paper investigates an intelligent reflecting surfaces (IRS) aided anti-jamming communication strategy in the integrated terrestrial-satellite network (ITSN), where the IRS is exploited to mitigate jamming interference and enhance the integrated system communication performance. In such a network, the terrestrial network and satellite network are co-existing with a spectrum-sharing scheme in the presence of a multi-antenna jammer. We aim at maximizing the weighted sum rate (WSR) of all users by jointly optimizing the terrestrial beamformers and IRS phase shifts while considering the signal-to-interference-plus-noise ratio (SINR) requirements of legitimate users. Different from the non-convex optimization techniques utilized in the IRS-related problem, a novel optimization-driven deep reinforcement learning (DRL) algorithm is proposed, which leverages both the robustness of model-free learning approaches and the efficiency of model-based optimization methods. In the optimization module of the proposed algorithm, we analyze the smart jammer under the unknown jamming model and derive a lower bound of the anti-jamming uncertainty, such that the IRS-aided anti-jamming problem can be solved by alteration method with second-order cone programming (SOCP) algorithm and semidefinite relaxation (SDR) technique. Simulation results demonstrate that the IRS can enhance the anti-jamming performance efficiently, and the proposed optimization-driven DRL algorithm can improve both the learning rate and the system performance compared with existing solutions.

Following the success of Natural Language Processing (NLP) transformers pretrained via self-supervised learning, similar models have been proposed recently for speech processing such as Wav2Vec2, HuBERT and UniSpeech-SAT. An interesting yet unexplored area of application of these models is Spoken Dialogue Systems, where the users’ audio signals are typically just mapped to word-level features derived from an Automatic Speech Recogniser (ASR), and then processed using NLP techniques to generate system responses. This paper reports a comprehensive comparison of dialogue policies trained using ASR-based transcriptions and extended with the aforementioned audio processing transformers in the DSTC2 task. Whilst our dialogue policies are trained with supervised and policy-based deep reinforcement learning, they are assessed using both automatic task completion metrics and a human evaluation. Our results reveal that using audio embeddings is more beneficial than detrimental in most of our trained dialogue policies, and that the benefits are stronger for supervised learning than reinforcement learning.

As a growing number of warehouse operators are moving from human-only to Collaborative human-robot Order Picking solutions, more efficient picker routing policies are needed, since the complexity of coordinating multiple actors in the system increases significantly. The objective of these policies is to match human pickers and robot carriers to fulfill picking tasks, optimizing pick-rate and total tardiness of the orders. In this paper, we propose to formulate the order picking routing problem as a more general combinatorial optimization problem known as Two-sided Online Bipartite Matching. We present an end-to-end Deep Reinforcement Learning approach to optimize a combination of pick-rate and order tardiness, and to deal with the uncertainty of real-world warehouse environments. To extract and exploit spatial information from the environment, we devise three different Graph Neural Network architectures and empirically evaluate them on several scenarios of growing complexity in a simulation environment we developed. We show that all proposed methods significantly outperform greedy and more sophisticated heuristics, as well as non-GNN-based DRL approaches. Moreover, our methods exhibit good transferability properties, even when scaling up test problem instances to more than forty times the size of the ones the models were trained on.