Search

We consider the problem of solving hybrid discrete-continuous Markov Decision Processes (MDPs) that are often encountered in computing optimal policies for complex multi-agent missions with both continuous vehicle dynamics and discrete mission-state transition models, in the presence of potential health degradations and failures of individual agents. A comprehensive Health Aware Planning (HAP) framework is proposed that establishes a feedback between mission planning and vehicle-level learning-focused adaptive controllers through online learned own models of agent health and capabilities. The HAP framework accounts for predicted likelihood of vehicle health degradations captured through probabilistic state-dependent models that are integrated into the MDP formulation. This proactive ability to anticipate health degradation and plan accordingly enables the HAP approach to consistently outperform planners that change the policies only after failures have occurred (reactive planners). The approach is tested on a large-scale (≈ 1010 state–action pairs) long-duration (persistent) target tracking scenario using a novel on-trajectory planning algorithm, and demonstrated to sustain higher mission performance by reducing the number of failures and re-assessing Unmanned Aerial Vehicle (UAV) capabilities.

This paper presents the development and hardware implementation of an autonomous battery maintenance mechatronic system that significantly extends the operational time of battery powered small-scaled unmanned aerial vehicles (UAVs). A simultaneous change and charge approach is used to overcome the significant downtime experienced by existing charge-only approaches. The automated system quickly swaps a depleted battery of a UAV with a replenished one while simultaneously recharging several other batteries. This results in a battery maintenance system with low UAV downtime, arbitrarily extensible operation time, and a compact footprint. Hence, the system can enable multi-agent UAV missions that require persistent presence. This capability is illustrated by developing and testing in flight a centralized autonomous planning and learning algorithm that incorporates a probabilistic health model dependent on vehicle battery health that is updated during the mission, and replans to improve the performance based on the improved model. Flight test results are presented for a 3-h-long persistent mission with three UAVs that each has an endurance of 8-10 min on a single battery charge (more than 100 battery swaps).

Interest in control of multiple autonomous vehicles continues to grow for applications such as weather monitoring, geographical mapping fauna surveys, and extra-terrestrial exploration. The task of persistent surveillance is of particular significance in that the target area needs to be continuously surveyed, minimizing the time between visitations to the same region. This distinction from one-time coverage does not allow a straightforward application of most exploration techniques to the problem, though ideas from these methods can still be used. The aerial vehicle dynamic and endurance constraints add additional complexity to the autonomous control problem, whereas stochastic environments and vehicle failures introduce uncertainty. In this work, we investigate techniques for high-level control, that are scalable, reliable, efficient, and robust to problem dynamics. Next, we suggest a modification to the control policy to account for aircraft dynamic constraints. We also devise a health monitoring policy and a control policy modification to improve performance under endurance constraints. The Vehicle Swarm Technology Laboratory-a hardware testbed developed at Boeing Research and Technology, Seattle, WA, for evaluating a swarm of unmanned air vehicles-is then described, and these control policies are tested in a realistic scenario.

To investigate and develop unmanned vehicle systems technologies for autonomous multiagent mission platforms, we are using an indoor multivehicle testbed called real-time indoor autonomous vehicle test environment (RAVEN) to study long-duration multivehicle missions in a controlled environment. Normally, demonstrations of multivehicle coordination and control technologies require that multiple human operators simultaneously manage flight hardware, navigation, control, and vehicle tasking. However, RAVEN simplifies all of these issues to allow researchers to focus, if desired, on the algorithms associated with high-level tasks. Alternatively, RAVEN provides a facility for testing low-level control algorithms on both fixed- and rotary-wing aerial platforms. RAVEN is also being used to analyze and implement techniques for embedding the fleet and vehicle health state (for instance, vehicle failures, refueling, and maintenance) into UAV mission planning. These characteristics facilitate the rapid prototyping of new vehicle configurations and algorithms without requiring a redesign of the vehicle hardware. This article describes the main components and architecture of RAVEN and presents recent flight test results illustrating the applications discussed above.

Currently, most fields in Western Siberia are classified as mature with significant reductions in reservoir pressure. This fact greatly complicates the drilling of horizontal wells in these fields. The role of quality and the requirements for drilling fluids increase significantly under such circumstances. Standard drilling fluids often fall short in achieving the drilling objectives because of significant overburdening of formations. To resolve this issue, the geological information provided by an operator drilling in Western Siberia was used as the basis for developing an oil-in-water direct emulsion fluid system that reduces the likelihood of formation damage and allows trouble-free drilling under abnormally low reservoir pressure (ALRP) conditions. Filtration studies were carried out, and the impact on the restoration of the core's natural permeability was evaluated before this fluid system was used in the field. Active use of this drilling mud system began in Western Siberia in 2002, with the drilling of horizontal wells. In their development, all wells showed much higher completion quality: an easy transition of the well into production mode without additional measures (swabbing, stimulation, etc.) and obtained production rates above those planned. The use of this type of drilling mud under ALRP conditions was accompanied by a substantial reduction in the risk of differential sticking, which in turn reduced the likelihood of losing expensive downhole equipment used in logging while drilling (LWD), measurement while drilling (MWD), and annular pressure while drilling (APWD), etc. Additionally, by aiding the maintenance of a minimum overburdening of formations, this fluid allows the use of conventional drilling equipment that does not require the time and cost of rig reconstruction, as is required in the case of oil-based mud (OBM). Based on the analysis of drilling efficiency and completion quality conducted after testing the use of the emulsion fluid to drill horizontal wells in reservoirs with ALRP, the research institute of the field operator has recommended this technology for regular use, especially for reservoirs with low porosity and permeability properties. The use of oil-in-water direct emulsion fluid is not intended as a replacement for the conventional water-based solutions used for reservoir completion in the cases of normal reservoir pressure values and in partially depleted reservoirs. However, where the reservoir conditions are such that the use of conventional water-based mud (WBM) is inefficient, the use of oil-in-water emulsion-based mud can substantially enhance drilling efficiency and completion quality.

In persistent missions, taking system’s health and capability degradation into account is an essential factor to predict and avoid failures. The state space in health-aware planning problems is often a mixture of continuous vehicle-level and discrete mission-level states. This in particular poses a challenge when the mission domain is partially observable and restricts the use of computationally expensive forward search methods. This paper presents a method that exploits a structure that exists in many health-aware planning problems and performs a two-layer planning scheme. The lower layer exploits the local linearization and Gaussian distribution assumption over vehicle-level states while the higher layer maintains a non-Gaussian distribution over discrete mission-level variables. This two-layer planning scheme allows us to limit the expensive online forward search to the mission-level states, and thus predict system’s behavior over longer horizons in the future. We demonstrate the performance of the method on a long duration package delivery mission using a quadrotor in a partially-observable domain in the presence of constraints and health/capability degradation.

This paper introduces a novel hierarchical decomposition approach for solving Multiagent Markov Decision Processes (MMDPs) by exploiting coupling relationships in the reward function. MMDP is a natural framework for solving stochastic multi-stage multiagent decision-making problems, such as optimizing mission performance of Unmanned Aerial Vehicles (UAVs) with stochastic health dynamics. However, computing the optimal solutions is often intractable because the state-action spaces scale exponentially with the number of agents. Approximate solution techniques do exist, but they typically rely on extensive domain knowledge. This paper presents the Hierarchically Decomposed MMDP (HD-MMDP) algorithm, which autonomously identifies different degrees of coupling in the reward function and decomposes the MMDP into a hierarchy of smaller MDPs that can be solved separately. Solutions to the smaller MDPs are embedded in an autonomously constructed tree structure to generate an approximate solution to the original problem. Simulation results show HD-MMDP obtains more cumulative reward than that of the existing algorithm for a ten-agent Persistent Search and Track (PST) mission, which is a cooperative multi-UAV mission with more than 10 19 states, stochastic fuel consumption model, and health progression model.

In large-scale persistent missions, the vehicle capabilities and health often degrade over time. This paper presents a Health Aware Planning (HAP) Framework for long-duration complex UAV missions by establishing close feedback between the high-level planning based on Markov Decision Processes (MDP) and the execution level learning-focused adaptive controllers. This feedback enables the HAP framework to plan by anticipating the failures and reassessing vehicle capabilities after the failures. This proactive behavior allows for efficient replanning to account for changing capabilities. Simulations for a 4 UAV target tracking scenario is presented to demonstrate the effectiveness of the proactive replanning capability of the presented HAP framework.

We consider the problem of high-level learning and decision making to enable multi-agent teams to autonomously tackle complex, large-scale missions, over long time periods in the presence of actuator failures. Agent health, measured by the functionality of its subsystems such as actuators, can change over time in long-duration missions and may depend on environmental states. This variability in agent health leads to uncertainty that can lead to inefficient plans, and in some cases even mission failure. The joint learning-planing problem becomes particularly challenging in a heterogeneous team where each agent may have a different correlation between their individual states and the state of the environment. We present a learning based planning framework for heterogeneous multiagent missions with health uncertainty that uses online learned probabilistic models of agent health. A decentralized incremental Feature Dependency Discovery algorithm is developed to enable agents to collaborate to efficiently learn representations of the uncertainty models across heterogeneous agents. The learned models of actuator failures allow our approach to plan in anticipation of potential health degradation. We show through large-scale planning under uncertainty simulations and flight experiments with state-dependent actuator and fuel-burnrate uncertainty that our planning approach can outperform planners that do not account for heterogeneity between agents.

planning problems in presence of state-correlated uncertainty.An online learning and planning framework is used to address the problem of improving planner performance for missions with state-dependent uncertain agent health dynamics. The framework includes a previously introduced Decentralized Multi-agent Markov decision process (Dec-MMDP) as an online planning algorithm that is scalable in number of agents, and Incremental Feature Discovery (iFDD) which is a compact and fast learning algorithm for estimating parameters of a state-correlated uncertainty model. In combination, this architecture yield an integrated learning-planning algorithm where the planning performance improves as uncertainty is reduced through learning. The presented algorithms are validated in a persistent search and track scenario with a novel automated battery swapping/recharging system that enables the UAVs to collaboratively track targets over durations that are signicantly larger than individual vehicle endurance with a single battery. The results indicate that the architecture can be used as an computationally ecient solution to multi-agent uncertain cooperative planning problems.

This paper extends prior work on the persistent mission problem where real-time changes in agent capability are included in the problem formulation. Here, we couple the mission planner with a low-level adaptive controller in real-time to: (1) Provide robustness against actuator degradations and (2) Use parameters internal to the adaptive controller to provide valuable insight into the physical capabilities of the agent. These parameters, in conjunction with sensor health information, form a more complete measure of agent capability, which is used online and in forward planning to enable both reactive and proactive behavior. Flight results are presented for a persistent mission scenario where actuator degradations are induced to demonstrate: (1) The robustness of the composite adaptive controller and its successful integration with the agent-level health-monitoring and mission-level planning systems and (2) The reactive and proactive qualities of the planning system in persistently re-tasking agents under actuator and sensor health degradations.

This paper presents an extension of our previous work on the persistent surveillance problem. An extended problem formulation incorporates real-time changes in agent capabilities as estimated by an onboard health monitoring system in addition to the existing communication constraints, stochastic sensor failure and fuel flow models, and the basic constraints of providing surveillance coverage using a team of autonomous agents. An approximate policy for the persistent surveillance problem is computed using a parallel, distributed implementation of the approximate dynamic programming algorithm known as Bellman Residual Elimination. This paper also presents flight test results which demonstrate that this approximate policy correctly coordinates the team to simultaneously provide reliable surveillance coverage and a communications link for the duration of the mission and appropriately retasks agents to maintain these services in the event of agent capability degradation.

Searching a complex environment for a hidden target is a common problem encountered by many autonomous systems. Many modern autonomous systems use a hierarchical structure for mission management where different algorithms perform different tasks to give agents desired behavior. This work investigates a search policy that guarantees both an exhaustive search of the map and conflict free paths of all agents. Agents formulate control decisions for a fixed number of time steps using a modular algorithm that allows parameterizations of agent capabilities. High fidelity simulation and flight test data using multiple autonomous vehicles are used to verify and validate the algorithms in real time.

This paper presents an extended formulation of the persistent surveillance problem first proposed in [1]. The extended formulation incorporates new communication constraints and a stochastic sensor failure model, in addition to modeling stochastic fuel flow dynamics and the basic constraints of providing surveillance coverage using a team of unmanned vehicles. Using a parallel, distributed implementation of an approximate dynamic programming algorithm, an approximate policy for the persistent surveillance problem can be quickly computed. Simulation analysis of this policy indicates that it correctly coordinates the actions of the team of UAVs to simultaneously provide reliable surveillance coverage and communications over the course of the mission, and appropriately retasks UAVs to maintain these services in the event of sensor failures.

Coordination of multiple, autonomous unmanned aerial vehicles is a growing topic within the research community. Universities and industry players are investing in Multi-vehicle, unmanned testbeds in order to better understand the benets and logistics of vehicle coordination. Boeing operates an indoor ight facility to enable hardware-in-the-loop, System of Systems evaluations in utilizing a heterogeneous autonomous eet of air and ground vehicles. Aside from providing a cost eective means of evaluating new autonomous vehicle technologies, the facility includes an existing eet of heterogeneous helicopters that are used to test high level, health based mission management concepts. This paper discusses the control and health-based management concepts required for the safe, reliable and ecient operation of vehicles in the facility. Details and advantages of the modular facility hardware architecture, which is key to rapid integration of new vehicle concepts, is described. A rapid control law design process, suitable for all types of helicopters, is also discussed and the actual ight performance of all vehicles is presented. Finally, the safety and reliability benets of autonomous, health based vehicle monitoring and behavior are discussed.

Increased levels of vehicle collaboration and auton omy are seen as a means to reduce overall mission completion costs while expanding mi ssion capabilities and increasing mission assurance for complex coupled system of systems. Systems health management technologies have made rapid advances that enable systems to know their own condition and capabilities, thus creating the opportunity for unprecedented lev els of adaptive control, real-time reconfiguration, and mission contingency management. Multi-agent task allocation and mission managements systems must account for vehicle- and system-level health-related issues to ensure that these systems are reliable an d cost effective to operate. Boeing’s Vehicle Swarm Technology Lab (VSTL), established in 2004, includes a 100’x50’x20’ testbed equipped with a vision-based motion capture indoor localization system. The testbed provides a cost-effective rapid prototyping capabil ity for integrating health-based adaptive control of subsystems, vehicle, mission, and swarms to guarantee top-level system-of-systems performance metrics. The lab’s heterogeneous fleet includes over 20 heterogeneous air vehicles, including VTOL and fixed wing, along with their ground stations and communication links in addition to heterogeneous ground vehicles and wall climbing robots. This paper discusses the Boeing VSTL design and capabilities, including the indoor localization system, multi-vehicle command and control (C2) and operator interface, realtime virtual environment, and health-based adaptive behaviors. The lab supports rapid prototyping and exploration of various multi-vehicl e operational concept of operations and missions including persistent surveillance, area se arch and tracking, and high density air traffic management. Additionally, the lab supports experimentation tasks for many other platform configuration and collaborative air, groun d, space, and maritime autonomous system of systems concepts.

The use of health based information in mission plan ning offers the opportunity to significantly enhance overall mission assurance. D eveloping mission concepts, even at a simple level, requires coordination of multiple ass ets and determination of common interfaces suitable for heterogeneous fleets. For systems that are subject to real failures, simulation offers the challenges of developing real istic scenarios and realistic health emulation. An alternative explored here is the use of indoor, rapid prototyping labs for exploring larger scale, heterogeneous mission conce pts. Of particular interest are persistent missions were faults are a key driver in the aggreg ate mission performance. Results of flight tests with several different sample missions will b e presented. These missions range from non-cooperative to cooperative and include a range of tasks.

The objective of this paper is to present the application of a methodology for designing cooperative control laws that guarantees safe coordination of unmanned aerial vehicles (UAVs). In parallel, an optimal control law and an avoidance control law are designed for each UAV of the multi-vehicle system. These two control laws are combined, guaranteeing that each UAV arrives at their desired state and that their path remains collision free.

Unmanned aerial vehicles (UAVs) are well-suited to a wide range of mission scenarios, such as search and rescue, border patrol, and military surveillance. The complex and distributed nature of these missions often requires teams of UAVs to work together. Furthermore, overall mission performance can be strongly influenced by vehicle failures or degradations, so an autonomous mission system must account for the possibility of these anomalies if it is to maximize performance. This paper presents a general health management methodology for designing mission systems that can anticipate the negative effects of various types of anomalies on the future mission state and choose actions that mitigate those effects. The formulation is then specialized to the problem of providing persistent surveillance coverage using a group of UAVs, where uncertain fuel usage dynamics and strong interdependence effects between vehicles must be considered. Finally, the paper presents results showing that the health-aware persistent surveillance planner based on this formulation exhibits excellent performance in both simulated and real flight test experiments.

Modern aircraft are capable of generating and collecting massive quantities of data from flight recorders, maintenance reports, logistics, and mission-readiness reporting systems. Current aircraft system health management schemes are developed based on data sources consisting of the aircraft's system operational conditions or maintenance and repair actions; however, crucial information regarding flight condition and situational parameters are often disregarded or used in a limited sense for diagnostics due to data accessibility issues or what can be an overwhelming volume of time-series data that is collected from modern onboard aircraft data recorders. With improved access to multiple sources of aircraft data, diagnostics and prognostics capabilities may be improved allowing flight crews, engineering, maintenance, and logistics users of an interoperable data system to make more informed decisions. Improved knowledge of the actual aircraft condition, usage and component life monitoring can significantly reduce aircraft operations and support (O&S) costs. In addition, integrating multiple sources of aircraft data can reduce total ownership cost (TOC), enhance mission safety and improve aircraft performance. There is a need to develop a multi-source data integration architecture for analyzing vast quantities of disparate data obtained from both onboard and off-board data sources. The multi-source data integration will be able to accurately combine disparate data and establish interconnectivity between distinct data sources to significantly improve data interoperability across multiple data sources. This type of information fusion can serve as the basis for developing advanced diagnostic and prognostic algorithms for legacy, current, and next- generation commercial and military aircraft.

Many existing and proposed military satellite communication waveforms use spectrally confined modulations to increase the number of users that occupy a limited bandwidth. These modulations are non-constant modulus and require linear operation of the transmitters. Linearity requirements necessitate that amplifiers operate well below their saturated output power, which often requires devices larger in size and weight than acceptable for mobile terminals, such as those in vehicles. The Lincoln Laboratory Amplifier Testbed (LLAT) is a measurement platform designed to quantify the effects of amplifiers on spectrally confined waveforms. LLAT is capable of measuring amplifiers at millimeter-wavelengths with saturated output powers ranging from a few Watts to hundreds of Watts. To allow maximum flexibility in modem design, LLAT uses virtual modems running on a multiprocessor grid. The LLAT enables investigation of technology advances that may solve the size, weight and power constraints of small mobile terminals.

Coordinated multi-vehicle autonomous systems can provide incredible functionality, but off-nominal conditions and degraded system components can render this capability ineffective. This paper presents techniques to improve mission-level functional reliability through better system self-awareness and adaptive mission planning. In particular, we extend the traditional definition of health management, which has historically referred to the process of actively monitoring and managing vehicle sub-systems (e.g., avionics) in the event of component failures, to the context of multiple vehicle operations and autonomous multi-agent teams. In this case, health management information about each mission system component is used to improve the mission system's self-awareness and adapt vehicle, guidance, task and mission plans. This paper presents the theoretical foundations of our approach and recent experimental results on a new UAV testbed.

This paper presents the development and implementation of techniques used to manage autonomous unmanned aerial vehicles (UAVs) performing 24/7 persistent surveillance operations. Using an indoor flight testbed, flight test results are provided to demonstrate the complex issues encountered by operators and mission managers when executing an extended persistent surveillance operation in realtime. This paper presents mission health monitors aimed at identifying and improving mission system performance to avoid down time, increase mission system eciency and reduce operator loading. This paper discusses the infrastructure needed to execute an autonomous persistent surveillance operation and presents flight test results from one of our recent automated UAV recharging experiments. Using the RAVEN at MIT, we present flight test results from a 24 hr, fully-autonomous air vehicle flight-recharge test and an autonomous, multi-vehicle extended mission test using small, electric-powered air vehicles.

Methods of time-controlled optimal flight-trajectory generation are developed that include the effects of risk from a threat environment. Lateral and vertical algorithms are developed for military jet aircraft with the intent of near real-time application. Simple analytic functions are used for threat models, as the focus of this paper is on general problem formulation and not detailed solutions for specific threats. A constant altitude, lateral flight-trajectory generation method is developed that optimizes with respect to time, fuel, final position, and risk exposure. Existing vertical plane trajectory generation methods that use standard direct operating costs of time and fuel are modified to include the effect of risks. Singular perturbation methods are used to obtain reduced-order airplane models that allow static rather than dynamic optimization. Pontryagin's Minimum Principle is used with a Fibonacci search method to minimize the cost functional. Formulation and numerical results are presented for both the horizontal and vertical plane problems. N the 1970's, rising fuel prices and improved micropro- cessor capability made consideration of onboard flight- path optimization appealing as a way to reduce operating costs. More recently, increased air traffic and the associated control problems have shifted research emphasis toward four- dimensional trajectory optimization. The four-dimensional problem typically includes a cost of time that allows trajecto- ries to meet required times of destination arrival. The majority of research in this area has been directed toward commercial operations and involves using the energy state approximation for generating vertical flight paths. It has been shown1^ that singular perturbation theory (SPT) may be used successfully to reduce the order of this problem. A reduced-order, often static, optimization technique can then be employed that greatly reduces computational burden. In this paper, methods of trajectory generation are developed that expand the basic SPT techniques to address the needs of military operations. Trajectory generation will be a fundamental requirement for future military aircraft flight management systems. These systems will be required to take advantage of all available information to perform integrated task processing and reduce pilot workload. In addition, the systems should be able to provide updates at any time throughout a mission or at regular time intervals sufficient for threat avoidance. Much of the previous work in trajectory generation for military air- craft5"7 has concentrated on feasible directions algorithms that use dynamic programming. These methods tend to be computationally intense and, therefore, are not well suited for onboard applications in dynamic threat environments. An ideal flight trajectory for military operations meets the mission requirements within the constraints of the aircraft limitations while minimizing exposure to threats. Several lev- els of information are used in the selection of such a strategic flight path and velocity profile. The trajectory will be a function of mission requirements (time of arrival, point of arrival, etc.), aircraft performance limitations (for example, fuel quantity and thrust limits), and the threat environment. The threat models used in this study do not refer to any specific military scenario. They are simple analytic functions

This paper extends the consensus-based bundle algorithm (CBBA), a distributed task allocation framework previously developed by the authors, to address complex missions for a team of heterogeneous agents in a dynamic environment. The extended algorithm proposes appropriate handling of time windows of validity for tasks, fuel costs of the vehicles, and heterogeneity in the agent capabilities, while preserving the robust convergence properties of the original algorithm. An architecture to facilitate real-time task replanning in a dynamic environment is also presented, along with methods to handle varying communication constraints and dynamic network topologies. Simulation results and experimental flight tests in an indoor test environment verify the proposed task planning methodology for complex missions.

This paper introduces a hardware platform for automated battery changing and charging for multiple UAV agents. The automated station holds a bu er of 8 batteries in a novel dual-drum structure that enables a "hot" battery swap, thus allowing the vehicle to remain powered on throughout the battery changing process. Each drum consists of four battery bays, each of which is connected to a smart-charger for proper battery maintenance and charging. The hot-swap capability in combination with local recharging and a large 8-battery capacity allow this platform to refuel multiple UAVs for long-duration and persistent missions with minimal delays and no vehicle shutdowns. Experimental results from the RAVEN indoor flight test facility are presented that demonstrate the capability and robustness of the battery change/charge station in the context of a multi-agent, persistent mission where surveillance is continuously required over a speci ed region., Boeing Scientific Research Laboratories, United States. Air Force Office of Scientific Research (FA9550-09-1-0522)