Search

We study the problem of inspection and maintenance planning of capital goods based on observations of the capital good's degradation state. However, the observations are imprecise, and their quality depends on the environment. For example, when performing maintenance for Heating, Ventilation, and Air-Conditioning units (HVACs) in trains, the health of the cooling component of an HVAC can be assessed from temperature readouts of the car in which the HVAC is mounted. Temperature information is useful in the summer when high car temperatures can indicate a failed cooling component, but this information has limited value during the winter. We model the problem as a partially observable Markov decision process with a fully observed environment. We analytically show that an environment-dependent monotonic at-most-4-region policy is optimal. Furthermore, we numerically analyze an example motivated by HVAC maintenance at Dutch Railways. This analysis shows that, in many cases, including the environment in the model can lead to cost savings of more than 10%. In a broad numerical experiment, we show that a simple policy cannot always substitute an optimal policy. [ABSTRACT FROM AUTHOR]

In this paper, we present a novel method to enhance the sum-rate effectiveness in full-duplex unmanned aerial vehicle (UAV)-assisted communication networks. Existing approaches often couple uplink and downlink associations, resulting in suboptimal performance, particularly in dynamic environments where user demands and network conditions are unpredictable. To overcome these limitations, we propose a decoupling of uplink and downlink associations for ground-based users (GBUs), significantly improving network efficiency. We formulate a comprehensive optimization problem that integrates UAV trajectory design and user association, aiming to maximize the overall sum-rate efficiency of the network. Due to the problem's non-convexity, we reformulate it as a Partially Observable Markov Decision Process (POMDP), enabling UAVs to make real-time decisions based on local observations without requiring complete global information. Our framework employs multi-agent deep reinforcement learning (MADRL), specifically the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm, which balances centralized training with distributed execution. This allows UAVs to efficiently learn optimal user associations and trajectory controls while dynamically adapting to local conditions. The proposed solution is particularly suited for critical applications such as disaster response and search and rescue missions, highlighting the practical significance of utilizing UAVs for rapid network deployment in emergencies. By addressing the limitations of existing centralized and distributed solutions, our hybrid model combines the benefits of centralized training with the adaptability of distributed inference, ensuring optimal UAV operations in real-time scenarios. [ABSTRACT FROM AUTHOR]

Unmanned aerial vehicles (UAVs) assisted Internet of things (IoT) data collection is an efficient and promising approach. The optimization of resource allocation in path planning is addressed in this paper by refining the energy consumption model and considering three metrics: the amount of collected data, time efficiency, and energy efficiency. The problem is formulated as a distributed partially observable Markov decision process (POMDP) and a novel deep reinforcement learning algorithm called RISE (Rényi state entropy)-D3QN (dueling double deep Q network) is proposed. It combines intrinsic rewards, prioritized experience replay, and soft-max exploration strategy, enabling path planning for UAV swarms while adapting to changes in UAV battery capacity, IoT device locations, data volume, and quantity. Simulation results demonstrate that compared to traditional D3QN and DQN algorithms, the proposed approach significantly increases. the amount of collected data from IoT devices while reducing UAV flight time and energy consumption, all while ensuring UAV safety during flight. [ABSTRACT FROM AUTHOR]

Robotic manipulators are widely used in various industries for complex and repetitive tasks. However, they remain vulnerable to unexpected hardware failures. In this study, we address the challenge of enabling a robotic manipulator to complete tasks despite joint malfunctions. Specifically, we develop a reinforcement learning (RL) framework to adaptively compensate for a nonfunctional joint during task execution. Our experimental platform is the Franka robot with seven degrees of freedom (DOFs). We formulate the problem as a partially observable Markov decision process (POMDP), where the robot is trained under various joint failure conditions and tested in both seen and unseen scenarios. We consider scenarios where a joint is permanently broken and where it functions intermittently. Additionally, we demonstrate the effectiveness of our approach by comparing it with traditional inverse kinematics-based control methods. The results show that the RL algorithm enables the robot to successfully complete tasks even with joint failures, achieving a high success rate with an average rate of 93.6%. This showcases its robustness and adaptability. Our findings highlight the potential of RL to enhance the resilience and reliability of robotic systems, making them better suited for unpredictable environments. [ABSTRACT FROM AUTHOR]

Reinforcement learning (RL) has garnered significant attention for developing decision-making agents that aim to maximize rewards, specified by an external supervisor, within fully observable environments. However, many real-world problems involve partial or noisy observations, where agents cannot access complete and accurate information about the environment. These problems are commonly formulated as partially observable Markov decision processes (POMDPs). Previous studies have tackled RL in POMDPs by either incorporating the memory of past actions and observations or by inferring the true state of the environment from observed data. Nevertheless, aggregating observations and actions over time becomes impractical in problems with large decision-making time horizons and high-dimensional spaces. Furthermore, inference-based RL approaches often require many environmental samples to perform well, as they focus solely on reward maximization and neglect uncertainty in the inferred state. Active inference (AIF) is a framework naturally formulated in POMDPs and directs agents to select actions by minimizing a function called expected free energy (EFE). This supplies reward-maximizing (or exploitative) behavior, as in RL, with information-seeking (or exploratory) behavior. Despite this exploratory behavior of AIF, its use is limited to problems with small time horizons and discrete spaces due to the computational challenges associated with EFE. In this article, we propose a unified principle that establishes a theoretical connection between AIF and RL, enabling seamless integration of these two approaches and overcoming their limitations in continuous space POMDP settings. We substantiate our findings with rigorous theoretical analysis, providing novel perspectives for using AIF in designing and implementing artificial agents. Experimental results demonstrate the superior learning capabilities of our method compared to other alternative RL approaches in solving partially observable tasks with continuous spaces. Notably, our approach harnesses information-seeking exploration, enabling it to effectively solve reward-free problems and rendering explicit task reward design by an external supervisor optional. [ABSTRACT FROM AUTHOR]

Partially Observable Markov Decision Processes (POMDPs) can model complex sequential decision-making problems under stochastic and uncertain environments. A main reason hindering their broad adoption in real-world applications is the unavailability of a suitable POMDP model or a simulator thereof. Available solution algorithms, such as Reinforcement Learning (RL), typically benefit from the knowledge of the transition dynamics and the observation generating process, which are often unknown and non-trivial to infer. In this work, we propose a combined framework for inference and robust solution of POMDPs via deep RL. First, all transition and observation model parameters are jointly inferred via Markov Chain Monte Carlo sampling of a hidden Markov model, which is conditioned on actions, in order to recover full posterior distributions from the available data. The POMDP with uncertain parameters is then solved via deep RL techniques with the parameter distributions incorporated into the solution via domain randomization, in order to develop solutions that are robust to model uncertainty. As a further contribution, we compare the use of Transformers and long short-term memory networks, which constitute model-free RL solutions and work directly on the observation space, with an approach termed the belief-input method, which works on the belief space by exploiting the learned POMDP model for belief inference. We apply these methods to the real-world problem of optimal maintenance planning for railway assets and compare the results with the current real-life policy. We show that the RL policy learned by the belief-input method is able to outperform the real-life policy by yielding significantly reduced life-cycle costs. [ABSTRACT FROM AUTHOR]

Inducing learning strategies is a crucial component of intelligent tutoring systems. Previous research has predominantly focused on the induction of offline learning strategies. Although the existing offline learning strategy induction methods can also be used for real-time updates of learning strategies, their update efficiency is not high, making it difficult to capture the characteristics exhibited by learners during the learning process in a timely manner. With the superior performance of the Partially Observable Markov Decision Process (POMDP), this paper proposes a POMDP-based cognitive experience model, which can be quickly updated during interactions and enables the real-time induction of learning strategies by weighting the learning experiences of different learners. Experimental results demonstrate that the learning strategies induced by PCEM are more personalized and exhibit superior performance. [ABSTRACT FROM AUTHOR]

Aiming to improve the efficiency of the online process in path planning, a novel searching method is proposed based on environmental information analysis. Firstly, a search and rescue (SAR) environmental model and an unmanned ground vehicle (UGV) motion model are established according to the characteristics of a mining environment. Secondly, an online search area path-planning method is proposed based on the gray system theory and the reinforcement learning theory to handle multiple constraints. By adopting the multi-attribute intelligent (MAI) gray decision process, the action selection decision can be dynamically adjusted based on the current environment, ensuring the stable convergence of the model. Finally, experimental verification is conducted in different small-scale mine SAR simulation scenarios. The experimental results show that the proposed search planning method can capture the target in the search area with a smoother convergence effect and a shorter path length than other path-planning algorithms. [ABSTRACT FROM AUTHOR]

Motion planning under uncertainty is essential for reliable robot operation. Despite substantial advances over the past decade, the problem remains difficult for systems with complex dynamics. Most state-of-the-art methods perform search that relies on a large number of forward simulations. For systems with complex dynamics, this generally requires costly numerical integrations, which significantly slows down the planning process. Linearization-based methods have been proposed that can alleviate the above problem. However, it is not clear how linearization affects the quality of the generated motion strategy, and when such simplifications are admissible. To answer these questions, we propose a non-linearity measure, called Statistical-distance-based Non-linearity Measure (SNM), that can identify where linearization is beneficial and where it should be avoided. We show that when the problem is framed as the Partially Observable Markov Decision Process, the value difference between the optimal strategy for the original model and the linearized model can be upper-bounded by a function linear in SNM. Comparisons with an existing measure on various scenarios indicate that SNM is more suitable in estimating the effectiveness of linearization-based solvers. To test the applicability of SNM in motion planning, we propose a simple online planner that uses SNM as a heuristic to switch between a general and a linearization-based solver. Results on a car-like robot with second order dynamics and 4-DOFs and 7-DOFs torque-controlled manipulators indicate that SNM can appropriately decide if and when a linearization-based solver should be used. [ABSTRACT FROM AUTHOR]

A comprehensive review of recent advancements in applying deep reinforcement learning (DRL) to fluid dynamics problems is presented. Applications in flow control and shape optimization, the primary fields where DRL is currently utilized, are thoroughly examined. Moreover, the review introduces emerging research trends in automation within computational fluid dynamics, a promising field for enhancing the efficiency and reliability of numerical analysis. Emphasis is placed on strategies developed to overcome challenges in applying DRL to complex, real-world engineering problems, such as data efficiency, turbulence, and partial observability. Specifically, the implementations of transfer learning, multi-agent reinforcement learning, and the partially observable Markov decision process are discussed, illustrating how these techniques can provide solutions to such issues. Finally, future research directions that could further advance the integration of DRL in fluid dynamics research are highlighted. [ABSTRACT FROM AUTHOR]

A main research goal in various studies is to use an observational data set and provide a new set of counterfactual guidelines that can yield causal improvements. Dynamic Treatment Regimes (DTRs) are widely studied to formalize this process and enable researchers to find guidelines that are both personalized and dynamic. However, available methods in finding optimal DTRs often rely on assumptions that are violated in real-world applications (e.g., medical decision making or public policy), especially when (a) the existence of unobserved confounders cannot be ignored, and (b) the unobserved confounders are time varying (e.g., affected by previous actions). When such assumptions are violated, one often faces ambiguity regarding the underlying causal model that is needed to be assumed to obtain an optimal DTR. This ambiguity is inevitable because the dynamics of unobserved confounders and their causal impact on the observed part of the data cannot be understood from the observed data. Motivated by a case study of finding superior treatment regimes for patients who underwent transplantation in our partner hospital (Mayo Clinic) and faced a medical condition known as new-onset diabetes after transplantation, we extend DTRs to a new class termed Ambiguous Dynamic Treatment Regimes (ADTRs), in which the causal impact of treatment regimes is evaluated based on a "cloud" of potential causal models. We then connect ADTRs to Ambiguous Partially Observable Markov Decision Processes (APOMDPs) proposed by Saghafian (2018), and consider unobserved confounders as latent variables but with ambiguous dynamics and causal effects on observed variables. Using this connection, we develop two reinforcement learning methods termed Direct Augmented V-Learning (DAV-Learning) and Safe Augmented V-Learning (SAV-Learning), which enable using the observed data to effectively learn an optimal treatment regime. We establish theoretical results for these learning methods, including (weak) consistency and asymptotic normality. We further evaluate the performance of these learning methods both in our case study (using clinical data) and in simulation experiments (using synthetic data). We find promising results for our proposed approaches, showing that they perform well even compared with an imaginary oracle who knows both the true causal model (of the data-generating process) and the optimal regime under that model. Finally, we highlight that our approach enables a two-way personalization; obtained treatment regimes can be personalized based on both patients' characteristics and physicians' preferences. This paper was accepted by David Simchi-Levi, data science. Supplemental Material: The data files and online appendix are available at https://doi.org/10.1287/mnsc.2022.00883. [ABSTRACT FROM AUTHOR]

Solving continuous Partially Observable Markov Decision Processes (POMDPs) is challenging, particularly for high-dimensional continuous action spaces. To alleviate this difficulty, we propose a new sampling-based online POMDP solver, called A daptive D iscretization using V oronoi T rees (ADVT). It uses Monte Carlo Tree Search in combination with an adaptive discretization of the action space as well as optimistic optimization to efficiently sample high-dimensional continuous action spaces and compute the best action to perform. Specifically, we adaptively discretize the action space for each sampled belief using a hierarchical partition called Voronoi tree, which is a Binary Space Partitioning that implicitly maintains the partition of a cell as the Voronoi diagram of two points sampled from the cell. ADVT uses the estimated diameters of the cells to form an upper-confidence bound on the action value function within the cell, guiding the Monte Carlo Tree Search expansion and further discretization of the action space. This enables ADVT to better exploit local information with respect to the action value function, allowing faster identification of the most promising regions in the action space, compared to existing solvers. Voronoi trees keep the cost of partitioning and estimating the diameter of each cell low, even in high-dimensional spaces where many sampled points are required to cover the space well. ADVT additionally handles continuous observation spaces, by adopting an observation progressive widening strategy, along with a weighted particle representation of beliefs. Experimental results indicate that ADVT scales substantially better to high-dimensional continuous action spaces, compared to state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Copyright of Journal of Chongqing University of Technology (Natural Science) is the property of Chongqing University of Technology and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Graph Neural Networks (GNNs) are susceptible to adversarial injection attacks, potentially compromising the model integrity, reducing accuracy, and posing security risks. However, most of the current countermeasures focus on enhancing the robustness of GNNs rather than directly addressing these specific attacks. The challenge stems from the difficulty of protecting all nodes in the entire graph and the agnostic of the attackers. Therefore, we propose a secure training strategy for GNNs that counters the vulnerability to adversarial injection attacks and overcomes the obstacle of partial observability in existing defense mechanisms—where defenders are only aware of the graph's post-attack structure and node attributes, without the identification of compromised nodes. Our strategy not only protects specific nodes but also extends security to all nodes in the graph. We model the graph security issues as a Partially Observable Markov Decision Process (POMDP) and use Graph Convolutional Memory (GCM) to transform the observations of a POMDP into states with temporal memory proceeding to use reinforcement learning to solve for the optimal defensive strategy. Finally, we prevent learning from malicious nodes by limiting the convolutional scope, thus defending against adversarial injection attacks. Our defense method is evaluated on five datasets, achieving an accuracy range of 74% to 86.7%, which represents an enhancement of approximately 5.09% to 100.26% over post-attack accuracies. Compared with various traditional experimental models, our method shows an accuracy improvement ranging from 0.82% to 100.26%. [ABSTRACT FROM AUTHOR]

The synthesis problem for partially observable Markov decision processes (POMDPs) is to compute a policy that provably adheres to one or more specifications. Yet, the general problem is undecidable, and policies require full (and thus potentially unbounded) traces of execution history. To provide good approximations of such policies, POMDP agents often employ randomization over action choices. We consider the problem of computing simpler policies for POMDPs, and provide several approaches to still ensure their expressiveness. Key aspects are (1) the combination of an arbitrary number of specifications the policies need to adhere to, (2) a restricted form of randomization, and (3) a light-weight preprocessing of the POMDP model to encode memory. We provide a novel encoding as a mixed-integer linear program as baseline to solve the underlying problems. Our experiments demonstrate that the policies we obtain are more robust, smaller, and easier to implement for an engineer than those obtained from state-of-the-art POMDP solvers. [ABSTRACT FROM AUTHOR]

In recent years, space research has shifted heavily its focus towards enhanced autonomy on-board spacecrafts for on-orbit servicing activities (OOS). OOS and proximity operations include a variety of activities: the focal point of this work is the autonomous guidance of a chaser spacecraft for the shape reconstruction of an artificial uncooperative object. Adaptive guidance depends on the ability of the system to build a map of the uncertain environment, figuring out its location inside of it and accordingly determining the control law. Thus, autonomous navigation is framed as an active Simultaneous Localization and Mapping (SLAM) problem and modeled as a Partially Observable Markov Decision Process (POMDP). A state-of-the-art Deep Reinforcement Learning (DRL) method, Proximal Policy Optimization (PPO), is investigated to develop an agent capable of cleverly planning the shape reconstruction of the target. Starting from previous research on the topic, this work develops further proposing a continuous action space, such that the agent is no more forced to choose between a predefined set of possible discrete actions, fixed both in magnitude and direction. In this way any combination of the three-dimensional thrust vector components is available. The chaser spacecraft is a small satellite mounting an electric propulsion engine defining the action space range, in linearized eccentric relative motion with the selected uncooperative object. Through a rendered triangular mesh, the agent capabilities of geometry reconstruction and mapping are evaluated, considering the number of quality pictures made for each face. Extensive training tests are performed with random initial conditions to verify the generalizing capability of the DRL agent. The results are then validated in a comprehensive testing campaign, whose primary focus is the introduction of noisy measurements coming from navigation, affecting pose estimation. The sensitivity of the proposed method to this condition is analyzed and the efficiency of a retraining procedure is examined. The applicability of DRL methods and neural networks to support autonomous guidance in a close proximity scenario is corroborated and the technique employed is vastly tested and verified. [ABSTRACT FROM AUTHOR]

Appropriately calibrated human trust is essential for successful Human-Agent collaboration. Probabilistic frameworks using a partially observable Markov decision process (POMDP) have been previously employed to model the trust dynamics of human behavior, optimising the outcomes of a task completed with a collaborative recommender system. A POMDP model utilising signal detection theory to account for latent user trust is presented, with the model working to calibrate user trust via the implementation of three distinct agent features: disclaimer message, request for additional information, and no additional feature. A simulation experiment is run to investigate the efficacy of the proposed POMDP model compared against a random feature model and a control model. Evidence demonstrates that the proposed POMDP model can appropriately adapt agent features in-task based on human trust belief estimates in order to achieve trust calibration. Specifically, task accuracy is highest with the POMDP model, followed by the control and then the random model. This emphasises the importance of trust calibration, as agents that lack considered design to implement features in an appropriate way can be more detrimental to task outcome compared to an agent with no additional features. [ABSTRACT FROM AUTHOR]

Purpose: Because of the various advantages of reinforcement learning (RL) mentioned above, this study uses RL to train unmanned aerial vehicles to perform two tasks: target search and cooperative obstacle avoidance. Design/methodology/approach: This study draws inspiration from the recurrent state-space model and recurrent models (RPM) to propose a simpler yet highly effective model called the unmanned aerial vehicles prediction model (UAVPM). The main objective is to assist in training the UAV representation model with a recurrent neural network, using the soft actor-critic algorithm. Findings: This study proposes a generalized actor-critic framework consisting of three modules: representation, policy and value. This architecture serves as the foundation for training UAVPM. This study proposes the UAVPM, which is designed to aid in training the recurrent representation using the transition model, reward recovery model and observation recovery model. Unlike traditional approaches reliant solely on reward signals, RPM incorporates temporal information. In addition, it allows the inclusion of extra knowledge or information from virtual training environments. This study designs UAV target search and UAV cooperative obstacle avoidance tasks. The algorithm outperforms baselines in these two environments. Originality/value: It is important to note that UAVPM does not play a role in the inference phase. This means that the representation model and policy remain independent of UAVPM. Consequently, this study can introduce additional "cheating" information from virtual training environments to guide the UAV representation without concerns about its real-world existence. By leveraging historical information more effectively, this study enhances UAVs' decision-making abilities, thus improving the performance of both tasks at hand. [ABSTRACT FROM AUTHOR]

Real-time bidding has become a major means for online advertisement exchange. The goal of a real-time bidding strategy is to maximize the benefits for stakeholders, e.g., click-through rates or conversion rates. However, in practise, the optimal bidding strategy for real-time bidding is constrained by at least three aspects: cost-effectiveness, the dynamic nature of market prices, and the issue of missing bidding values. To address these challenges, we propose Imagine and Imitate Bidding (IIBidder), which includes Strategy Imitation and Imagination modules, to generate cost-effective bidding strategies under partially observable price landscapes. Experimental results on the iPinYou and YOYI datasets demonstrate that IIBidder reduces investment costs, optimizes bidding strategies, and improves future market price predictions. [ABSTRACT FROM AUTHOR]

In real-world scenarios, making navigation decisions for autonomous driving involves a sequential set of steps. These judgments are made based on partial observations of the environment, while the underlying model of the environment remains unknown. A prevalent method for resolving such issues is reinforcement learning, in which the agent acquires knowledge through a succession of rewards in addition to fragmentary and noisy observations. This study introduces an algorithm named deep reinforcement learning navigation via decision transformer (DRLNDT) to address the challenge of enhancing the decision-making capabilities of autonomous vehicles operating in partially observable urban environments. The DRLNDT framework is built around the Soft Actor-Critic (SAC) algorithm. DRLNDT utilizes Transformer neural networks to effectively model the temporal dependencies in observations and actions. This approach aids in mitigating judgment errors that may arise due to sensor noise or occlusion within a given state. The process of extracting latent vectors from high-quality images involves the utilization of a variational autoencoder (VAE). This technique effectively reduces the dimensionality of the state space, resulting in enhanced training efficiency. The multimodal state space consists of vector states, including velocity and position, which the vehicle's intrinsic sensors can readily obtain. Additionally, latent vectors derived from high-quality images are incorporated to facilitate the Agent's assessment of the present trajectory. Experiments demonstrate that DRLNDT may achieve a superior optimal policy without prior knowledge of the environment, detailed maps, or routing assistance, surpassing the baseline technique and other policy methods that lack historical data. [ABSTRACT FROM AUTHOR]

Information‐driven mission abort is an effective way to control the failure risk of safety‐critical systems during mission executions. We investigate the optimal sampling and mission abort decisions of partially observable safety‐critical systems, where the underlying system health state can only be revealed by sampling. In contrast to previous studies, we employ partial health information to jointly determine: (a) whether to execute sampling, and (b) when to abort the mission in a dynamic manner, so as to minimize the expected total cost incurred by sampling, mission failure, and system malfunction. Dynamic sampling and mission abort policies are devised following the belief state, whose optimization model is cast into the framework of a partially observable Markov decision process. Some structural insights with regard to the value function, control limit selection, and optimality existence are presented. The performance of the proposed sampling and abort policy is tested by numerical experiments, which are proved to outperform other heuristic abort policies in mission loss control. [ABSTRACT FROM AUTHOR]

The intermittent renewable energy in a virtual power plant (VPP) brings generation uncertainties, which prevents the VPP from providing a reliable and user-friendly power supply. To address this issue, this paper proposes a gated recurrent unit proximal policy optimization (GRUPPO)-based optimal VPP economic dispatch method. First, electrical generation, storage, and consumption are established to form a VPP framework by considering the accessibility of VPP state information. The optimal VPP economic dispatch can then be expressed as a partially observable Markov decision process (POMDP) problem. A novel deep reinforcement learning method called GRUPPO is further developed based on VPP time series characteristics. Finally, case studies are conducted over a 24-h period based on the actual historical data. The test results illustrate that the proposed economic dispatch can achieve a maximum operation cost reduction of 6.5% and effectively smooth the supply-demand uncertainties. [ABSTRACT FROM AUTHOR]

Target assignment and path planning are crucial for the cooperativity of multiple unmanned aerial vehicles (UAV) systems. However, it is a challenge considering the dynamics of environments and the partial observability of UAVs. In this article, the problem of multi-UAV target assignment and path planning is formulated as a partially observable Markov decision process (POMDP), and a novel deep reinforcement learning (DRL)-based algorithm is proposed to address it. Specifically, a target assignment network is introduced into the twin-delayed deep deterministic policy gradient (TD3) algorithm to solve the target assignment problem and path planning problem simultaneously. The target assignment network executes target assignment for each step of UAVs, while the TD3 guides UAVs to plan paths for this step based on the assignment result and provides training labels for the optimization of the target assignment network. Experimental results demonstrate that the proposed approach can ensure an optimal complete target allocation and achieve a collision-free path for each UAV in three-dimensional (3D) dynamic multiple-obstacle environments, and present a superior performance in target completion and a better adaptability to complex environments compared with existing methods. [ABSTRACT FROM AUTHOR]

Autonomous Unmanned Aerial Vehicles (UAVs) have possible applications in wildlife monitoring, disaster monitoring, and emergency Search and Rescue (SAR). Autonomous capabilities such as waypoint flight modes and obstacle avoidance, as well as their ability to survey large areas, make UAVs the prime choice for these critical applications. However, autonomous UAVs usually rely on the Global Navigation Satellite System (GNSS) for navigation and normal visibility conditions to obtain observations and data on their surrounding environment. These two parameters are often lacking due to the challenging conditions in which these critical applications can take place, limiting the range of utilisation of autonomous UAVs. This paper presents a framework enabling a UAV to autonomously navigate and detect targets in GNSS-denied and visually degraded environments. The navigation and target detection problem is formulated as an autonomous Sequential Decision Problem (SDP) with uncertainty caused by the lack of the GNSS and low visibility. The SDP is modelled as a Partially Observable Markov Decision Process (POMDP) and tested using the Adaptive Belief Tree (ABT) algorithm. The framework is tested in simulations and real life using a navigation task based on a classic SAR operation in a cluttered indoor environment with different visibility conditions. The framework is composed of a small UAV with a weight of 5 kg, a thermal camera used for target detection, and an onboard computer running all the computationally intensive tasks. The results of this study show the robustness of the proposed framework to autonomously explore and detect targets using thermal imagery under different visibility conditions. Devising UAVs that are capable of navigating in challenging environments with degraded visibility can encourage authorities and public institutions to consider the use of autonomous remote platforms to locate stranded people in disaster scenarios. [ABSTRACT FROM AUTHOR]

Active non-cooperative target tracking is an important ability for free-floating space manipulator (FFSM) systems. Aiming at the difficulty of pose and posture estimation of non-cooperative targets and the complexity of FFSM planning and control, the paper proposed an end-to-end target tracking algorithm using deep reinforcement learning (DRL) algorithm. The algorithm uses raw camera images as input and directly outputs the manipulator joint velocities. To deal with the partially observable Markov decision processes (POMDP) problem caused by single image input, we combine the soft actor–critic algorithm with recurrent neural networks. Therefore, the velocity information of the target is introduced to transform the task into a fully observable Markov decision-making process (MDP). Our approach is completely data-driven. It avoids the position and posture estimation of non-cooperative target and the complex modeling motion planning process of FFSM. We trained and tested the algorithm by building a simulation environment in CoppeliaSim. The results showed that our approach achieved good results in active non-cooperative target tracking of FFSM systems and demonstrated the great potential of DRL algorithms in solving space tasks. [ABSTRACT FROM AUTHOR]

Partially Observable Markov Decision Processes (POMDPs) are a powerful framework for planning under uncertainty. They allow to model state uncertainty as a belief probability distribution. Approximate solvers based on Monte Carlo sampling show great success to relax the computational demand and perform online planning. However, scaling to complex realistic domains with many actions and long planning horizons is still a major challenge, and a key point to achieve good performance is guiding the action-selection process with domain-dependent policy heuristics which are tailored for the specific application domain. We propose to learn high-quality heuristics from POMDP traces of executions generated by any solver. We convert the belief-action pairs to a logical semantics, and exploit data- and time-efficient Inductive Logic Programming (ILP) to generate interpretable belief-based policy specifications, which are then used as online heuristics. We evaluate thoroughly our methodology on two notoriously challenging POMDP problems, involving large action spaces and long planning horizons, namely, rocksample and pocman. Considering different state-of-the-art online POMDP solvers, including POMCP, DESPOT and AdaOPS, we show that learned heuristics expressed in Answer Set Programming (ASP) yield performance superior to neural networks and similar to optimal handcrafted taskspecific heuristics within lower computational time. Moreover, they well generalize to more challenging scenarios not experienced in the training phase (e.g., increasing rocks and grid size in rocksample, incrementing the size of the map and the aggressivity of ghosts in pocman). [ABSTRACT FROM AUTHOR]

Civil and maritime engineering systems must be efficiently managed to control the failure risk at an acceptable level as their performance is gradually degraded throughout the operational life, caused by fatigue and corrosion. Structural health monitoring develops a timely capability to assess the structural condition and performance metrics. However, using actual long-term monitoring data to guide the life-cycle management under stochastic environments has not been sufficiently studied. To realize an optimal maintenance strategy within the service life, an integrated monitoring-based optimal management framework is developed on the basis of the partially observable Markov decision processes (POMDPs) and Bayesian forecasting. In the proposed framework, the stochastic fatigue processes are quantified by the state transition matrix. The Bayesian dynamic linear model is embedded in POMDPs as a continuous observation part to forecast the cycling impacts and estimate the deterioration rate using long-term dynamic strain responses. In addition, making use of the special features of the problem considered in this paper, an adaptive discretization strategy is proposed to alleviate the complexity of large discrete observed spaces in the POMDP. The applicability and feasibility of the framework are evaluated by intelligent maintenance of fatigue-sensitive components with real-world monitoring data. After solving the POMDP by an efficient offline solver, the results obtained in this paper demonstrate that structural interventions are uneconomical to extend the life when a welded detail is approaching its end of life due to the normal service. Furthermore, if multiple interventions are available, the framework can find optimal maintenance actions based on the trade-off between long-term utility and the corresponding cost. This framework as the prototype could also be adjusted to aid life-cycle intelligent maintenance of other types of components under different deterioration scenarios. [ABSTRACT FROM AUTHOR]

An extended flexible job scheduling problem is presented with characteristics of technology and path flexibility (dual flexibility), varied transportation time, and an uncertain environment. The scheduling can greatly increase efficiency and security in complex scenarios, e.g., distributed vehicle manufacturing, and multiple aircraft maintenance. However, optimizing the scheduling puts forward higher requirements on accuracy, real time, and generalization, while subject to the curse of dimension and usually incomplete information. Various coupling relations among operations, stations, and resources aggravate the problem. To deal with the above challenges, we propose a multi-agent reinforcement learning algorithm where the scheduling environment is modeled as a decentralized partially observable Markov decision process. Each job is regarded as an agent that decides the next triplet, i.e., operation, station, and employed resource. This paper is novel in addressing the flexible job shop scheduling problem with dual flexibility and varied transportation time in consideration and proposing a double Q-value mixing (DQMIX) optimization algorithm under a multi-agent reinforcement learning framework. The experiments of our case study show that the DQMIX algorithm outperforms existing multi-agent reinforcement learning algorithms in terms of solution accuracy, stability, and generalization. In addition, it achieves better solution quality for larger-scale cases than traditional intelligent optimization algorithms. [ABSTRACT FROM AUTHOR]

Summary: With the rise of industrialization, the importance of the industrial Internet of Things (IIoT) has increased significantly, and with it comes a variety of security threats. Therefore, the security of these networks is critical. Industrial Response Systems (IRSs), as the last line of security, plays an important role in the security system of the Industrial Internet of Things. In this paper, a new IRS model based on the non‐cooperative game is proposed. First, by combining the Partially Observable Markov Decision Process (POMDP) model with the stochastic game model based on the expanded attack tree, our model could effectively perceive the changes at each node. Second, our model incorporates the alarms of intrusion detection system (IDS) and the physical quantities of sensors in Industrial Cyber‐Physical System (ICPS) into the quantization system so that the model can respond to intruders more accurately and comprehensively. Finally, we develop this model based on multiprocessors to speed up the solution process, and adopt an approximation algorithm to reduce the number of iterations of the POMDP [ABSTRACT FROM AUTHOR]

Aerial base stations (AeBS), as crucial components of air-ground integrated networks, can serve as the edge nodes to provide flexible services to ground users. Optimizing the deployment of multiple AeBSs to maximize system energy efficiency is currently a prominent and actively researched topic in the AeBS-assisted edge-cloud computing network. In this paper, we deploy AeBSs using multi-agent deep reinforcement learning (MADRL). We describe the multi-AeBS deployment challenge as a decentralized partially observable Markov decision process (Dec-POMDP), taking into consideration the constrained observation range of AeBSs. The hypergraph convolution mix deep deterministic policy gradient (HCMIX-DDPG) algorithm is designed to maximize the system energy efficiency. The proposed algorithm uses the value decomposition framework to solve the lazy agent problem, and hypergraph convolutional (HGCN) network is introduced to strengthen the cooperative relationship between agents. Simulation results show that the suggested HCMIX-DDPG algorithm outperforms alternative baseline algorithms in the multi-AeBS deployment scenario. [ABSTRACT FROM AUTHOR]

In modern electronic warfare, radar intelligence has become increasingly crucial when dealing with complex interference environments. This paper combines radar agile frequency technology with reinforcement learning to achieve adaptive frequency hopping for radar anti-jamming. Unlike traditional reinforcement learning with Markov decision processes (MDPs), the interaction between radar and jammers occurs within the partially observable Markov decision processes (POMDPs). In this context, the partial observation information available to the agent does not strictly satisfy the Markov property. This paper uses multiple layers of historical observation information to solve this problem. Historical observations can be viewed as a time series, and time-sensitive networks are employed to extract the temporal information embedded within the observations. In addition, the reward function is optimized to facilitate the faster learning of the agent in the jammer sweep environment. This simulation shows that the optimization of the agent state, network structure, and reward function can effectively help the radar to resist jamming. [ABSTRACT FROM AUTHOR]

A cognitive radar (CR) system can offer enhanced target tracking performance due to its intelligence on the perception‐action cycle, wherein a CR adaptively allocates the limited transmitting resources based on its perception of surrounding environments. To effectively manage the transmit waveform resource for the target tracking task, CR resource management problem is formulated under the partially observable Markov decision process framework. The sequential decision‐making and the inherent partial observability for target tracking problem are considered. In the proposed method, a long short‐term memory (LSTM)‐based twin delayed deep deterministic policy gradient (TD3) algorithm is developed to effectively solve the problem. A reward function is designed considering Haykin's cognitive executive attention mechanism for radar systems such that the CR resource management policy has stability in the decision of transmit waveform, which follows the principle of minimum disturbance. Simulation results demonstrate the superiority of the proposed LSTM memory‐based TD3 with improved target tracking performance and increased mean rewards for CR. [ABSTRACT FROM AUTHOR]

This paper investigates the optimality of the risk probability for finite horizon partially observable discrete-time Markov decision processes (POMDPs). The probability of the risk is optimized based on the criterion of total rewards not exceeding the preset goal value, which is different from the optimal problem of expected rewards. Based on the Bayes operator and the filter equations, the optimization problem of risk probability can be equivalently reformulated as filtered Markov decision processes. As an advantage of developing the value iteration technique, the optimality equation satisfied by the value function is established and the existence of the risk probability optimal policy is proven. Finally, an example is given to illustrate the effectiveness of using the value iteration algorithm to compute the value function and optimal policy. [ABSTRACT FROM AUTHOR]

The existing research on dependent task offloading and resource allocation assumes that edge servers can provide computational and communication resources free of charge. This paper proposes a two-stage resource allocation method to address this issue. In the first stage, users incentivize edge servers to provide resources. We formulate the incentive problem in this stage as a multivariate Stackelberg game, which takes into account both computational and communication resources. In addition, we also analyze the uniqueness of the Stackelberg equilibrium under information sharing conditions. Considering the privacy issues of the participants, the research is extended to scenarios without information sharing, where the multivariable game problem is modeled as a partially observable Markov decision process (POMDP). In order to obtain the optimal incentive decision in this scenario, a reinforcement learning algorithm based on the learning game is designed. In the second stage, we propose a greedy-based deep reinforcement learning algorithm that is aimed at minimizing task execution time by optimizing resource and task allocation strategies. Finally, the simulation results demonstrate that the algorithm designed for non-information sharing scenarios can effectively approximate the theoretical Stackelberg equilibrium, and its performance is found to be better than that of the other three benchmark methods. After the allocation of resources and sub-tasks by the greedy-based deep reinforcement learning algorithm, the execution delay of the dependent task is significantly lower than that in local processing. [ABSTRACT FROM AUTHOR]

Autonomous shuttles have been used as end-mile solutions for smart mobility in smart cities. The urban driving conditions of smart cities with many other actors sharing the road and the presence of intersections have posed challenges to the use of autonomous shuttles. Round intersections are more challenging because it is more difficult to perceive the other vehicles in and near the intersection. Thus, this paper focuses on the decision-making of autonomous vehicles for handling round intersections. The round intersection is introduced first, followed by introductions of the Markov Decision Process (MDP), the Partially Observable Markov Decision Process (POMDP) and the Object-Oriented Partially Observable Markov Decision Process (OOPOMDP), which are used for decision-making with uncertain knowledge of the motion of the other vehicles. The Partially Observable Monte-Carlo Planning (POMCP) algorithm is used as the solution method and OOPOMDP is applied to the decision-making of autonomous vehicles in round intersections. Decision-making is formulated first as a POMDP problem, and the penalty function is formulated and set accordingly. This is followed by an improvement in decision-making with policy prediction. Augmented objective state and policy-based state transition are introduced, and simulations are used to demonstrate the effectiveness of the proposed method for collision-free handling of round intersections by the ego vehicle. [ABSTRACT FROM AUTHOR]

An introduction is presented in which the editor discusses articles in the issue on topics including the development of a practical stochastic pricing model for multi-interval real-time markets, the rectangular decomposition of mixed-integer programs, and enhancing renewable energy integration.

In many real-life situations, the inventory record may not match the actual stock perfectly. This can happen due to distortion of inventory data, such as transaction errors, misplaced inventories, and spoilage. In these cases, because the decision maker only has incomplete information about the inventory levels, many well-known inventory policies are not even admissible, and our understanding of the optimal policies, even their existence, is very limited. In "Average Cost Optimality in Partially Observable Lost-Sales Inventory Systems," Bai et al. consider the classical lost-sales inventory model, in which the inventory level is only observed when it becomes zero. They formulate the cost-minimization problem as a partially observable Markov decision process. By exploiting the vanishing discount factor approach, they provide a way to verify the existence of optimal policies under the average cost criterion. The key step in their analysis is the construction of a valid policy, which, in a certain sense, copies the actions of another policy for the process starting from another initial state. We consider a partially observable lost-sales inventory system, in which the inventory level is observed only when it reaches zero. We use the vanishing discount factor approach to prove the existence of a stationary optimal policy for the average cost minimization. As our main methodological contribution, we provide a way to verify the key condition of the vanishing discount factor approach—the uniform boundedness of the relative discounted value function. To accomplish that, we construct a valid policy, which, in a certain sense, "copies" the actions of another policy for the process with a different initial state. To the best of our knowledge, this paper is the first one on partially observable inventory models under the average cost criterion. [ABSTRACT FROM AUTHOR]

In edge-enabled Internet of Things (IoT), Quality of Experience (QoE)-guaranteed offloading decision is to decide which IoT tasks can be offloaded to edge servers with QoE guarantee. Centralized QoE-guaranteed offloading decision methods construct a global decision model for all IoT tasks with complete information. However, centralized offloading decision methods entail collecting global information from IoT devices, edge servers, and network environment, which may not be practical in large-scale distributed edge-enabled IoT environments, and it is unrealistic for privacy-critical and heterogeneous IoT tasks in many real-world edge-enabled IoT systems, where IoT devices may refuse to expose their private information and heterogeneous IoT tasks may have different QoE requirements, these issues make the application of centralized offloading decision method limited. To address these limitations, we propose a distributed offloading decision method which enables each IoT device to make decisions by partially observable global information in a decentralized manner. The distributed offloading decision process is modeled as a multi-agent partially observable Markov decision process, and a novel model-free deep reinforcement learning-based distributed algorithm named GRU Fictitious Self-Play Dueling Double Deep Recurrent Q Network(GFSP-D3RQN) is introduced to solve the problem. Furthermore, we measure the QoE of each IoT device based on a combination of latency and energy consumption, which are weighted differently according to the individual preferences of each IoT device, using a non-dimensionalized adjustment to accommodate the varying requirements of these IoT devices. Extensive simulation results show that our algorithm can achieve a higher average QoE and higher success ratio compared with baseline algorithms, which improved by at least 6.38 % and 5.91 % , respectively. [ABSTRACT FROM AUTHOR]

Our goal is to develop theory and algorithms for establishing fundamental limits on performance imposed by a robot's sensors for a given task. In order to achieve this, we define a quantity that captures the amount of task-relevant information provided by a sensor. Using a novel version of the generalized Fano's inequality from information theory, we demonstrate that this quantity provides an upper bound on the highest achievable expected reward for one-step decision-making tasks. We then extend this bound to multi-step problems via a dynamic programming approach. We present algorithms for numerically computing the resulting bounds, and demonstrate our approach on three examples: (i) the lava problem from the literature on partially observable Markov decision processes, (ii) an example with continuous state and observation spaces corresponding to a robot catching a freely-falling object, and (iii) obstacle avoidance using a depth sensor with non-Gaussian noise. We demonstrate the ability of our approach to establish strong limits on achievable performance for these problems by comparing our upper bounds with achievable lower bounds (computed by synthesizing or learning concrete control policies). [ABSTRACT FROM AUTHOR]

In this paper, we prove that the optimal risk-sensitive reward for Markov decision processes with compact state space and action space converges to the optimal average reward as the risk-sensitive factor tends to 0. In doing so, a variational formula for the optimal risk-sensitive reward is derived. An extension of the Kreĭn-Rutman Theorem to certain nonlinear operators is involved. Based on these, partially observable Markov decision processes are also investigated. A portfolio optimization problem is presented as an example of an application of the approach, in which a duality-relation between the maximization of risk-sensitive reward and the maximization of upside chance for out-performance over the optimal average reward is established. [ABSTRACT FROM AUTHOR]

Constrained partially observable Markov decision processes (CPOMDPs) have been used to model various real-world phenomena. However, they are notoriously difficult to solve to optimality, and there exist only a few approximation methods for obtaining high-quality solutions. In this study, grid-based approximations are used in combination with linear programming (LP) models to generate approximate policies for CPOMDPs. A detailed numerical study is conducted with six CPOMDP problem instances considering both their finite and infinite horizon formulations. The quality of approximation algorithms for solving unconstrained POMDP problems is established through a comparative analysis with exact solution methods. Then, the performance of the LP-based CPOMDP solution approaches for varying budget levels is evaluated. Finally, the flexibility of LP-based approaches is demonstrated by applying deterministic policy constraints, and a detailed investigation into their impact on rewards and CPU run time is provided. For most of the finite horizon problems, deterministic policy constraints are found to have little impact on expected reward, but they introduce a significant increase to CPU run time. For infinite horizon problems, the reverse is observed: deterministic policies tend to yield lower expected total rewards than their stochastic counterparts, but the impact of deterministic constraints on CPU run time is negligible in this case. Overall, these results demonstrate that LP models can effectively generate approximate policies for both finite and infinite horizon problems while providing the flexibility to incorporate various additional constraints into the underlying model. [ABSTRACT FROM AUTHOR]

Copyright of Electric Power Automation Equipment / Dianli Zidonghua Shebei is the property of Electric Power Automation Equipment Press and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Patient‐reported outcomes (PROs) play an increasingly important role in medical decision making. Yet, patients whose objectives differ from their physician's may strategically report symptoms to alter treatment decisions. For example, athletes may underreport symptoms to expedite return‐to‐play (RTP) from sports‐related concussion (SRC). Thus, clinicians must implement treatment policies that mitigate the Price of Naïveté, that is, the reduction in health outcomes due to naïvely believing strategically reported symptoms. In this study, we analyze dynamic treatment cessation decisions with strategic patients. Specifically, we formulate the Behavior‐Aware Partially Observable Markov Decision Process (BA‐POMDP), which optimizes the timing of treatment cessation decisions while accounting for known symptom‐reporting behaviors. We then analytically characterize the BA‐POMDP's optimal policy, leading to several practical insights. Next, we formulate the Behavior‐Learning Partially Observable Markov Decision Process (BL‐POMDP), which extends the BA‐POMDP by learning a patient's symptom‐reporting behavior over time. We show that the BL‐POMDP is decomposable into several BA‐POMDPs, allowing us to leverage the BA‐POMDP's structural properties for solving the BL‐POMDP. Then, we apply the BL‐POMDP to RTP from SRC using data from 29 institutions across the United States. We estimate the Price of Naïveté by comparing the BL‐POMDP to naïve benchmark policies. Accordingly, the BL‐POMDP reduces premature RTP by over 44% and provides up to 3.63 additional health‐adjusted athletic exposures per athlete compared to current practice. Overall, changing the interpretation of reported symptoms can better reduce the Price of Naïveté over adjusting treatment cessation thresholds. Therefore, to improve patients' health outcomes, clinicians must understand how strategic behavior manifests in PROs. [ABSTRACT FROM AUTHOR]

Using reinforcement learning to control traffic signal systems has been discussed in recent years, but most works focused on simple scenarios such as a single crossroads, and the methods aiming at large-scale traffic scenarios face long-time training and suboptimal results. In this work, we develop a new multi-agent reinforcement model for large-scale traffic signal control tasks, and a curriculum transfer learning method is developed to optimize the joint policy step by step. The policies for different intersections are trained in a partially observable Markov decision process with centralized training and decentralized execution mechanism, and we design transformer modules for both the policy and evaluation networks by attention mechanism. We first train policies in a simple traffic scenario, and then these policies are transferred to the next curriculum by policy reloading, while the experiences of the source task are reused selectively. With the number of agents increasing, our method can achieve satisfactory performances quickly by reusing the knowledge from previous curriculums. We conduct several experiments on the Cityflow testbed. In the case of more than 10 crossroads, our model improve the mean reward from 3.0 to 5.0. [ABSTRACT FROM AUTHOR]

Autonomous systems that can assist humans with increasingly complex tasks are becoming ubiquitous. Moreover, it has been established that a human's decision to rely on such systems is a function of both their trust in the system and their own self-confidence as it relates to executing the task of interest. Given that both under- and over-reliance on automation can pose significant risks to humans, there is motivation for developing autonomous systems that could appropriately calibrate a human's trust or self-confidence to achieve proper reliance behavior. In this article, a computational model of coupled human trust and self-confidence dynamics is proposed. The dynamics are modeled as a partially observable Markov decision process without a reward function (POMDP/R) that leverages behavioral and self-report data as observations for estimation of these cognitive states. The model is trained and validated using data collected from 340 participants. Analysis of the transition probabilities shows that the proposed model captures the probabilistic relationship between trust, self-confidence, and reliance for all discrete combinations of high and low trust and self-confidence. The use of the proposed model to design an optimal policy to facilitate trust and self-confidence calibration is a goal of future work. [ABSTRACT FROM AUTHOR]