Search

Reinforcement learning (RL) can improve control performance by seeking to learn optimal control policies in the end-use environment for vehicles and other systems. To accomplish this, RL algorithms need to sufficiently explore the state and action spaces. This presents inherent safety risks, and applying RL on safety-critical systems like vehicle powertrain control requires safety enforcement approaches. In this paper, we seek control-barrier function (CBF)-based safety certificates that demarcate safe regions where the RL agent could optimize the control performance. In particular, we derive optimal high-order CBFs that avoid conservatism while ensuring safety for a vehicle in traffic. We demonstrate the workings of the high-order CBF with an RL agent which uses a deep actor-critic architecture to learn to optimize fuel economy and other driver accommodation metrics. We find that the optimized high-order CBF allows the RL-based powertrain control agent to achieve higher total rewards without any crashes in training and evaluation while achieving better accommodation of driver demands compared to previously proposed exponential barrier function filters and model-based baseline controllers.

Low ambient temperatures significantly reduce Lithium ion batteries' (LIBs') charge/discharge power and energy capacity, and cause rapid degradation through lithium plating. These limitations can be addressed by preheating the LIB with an external heat source or by exploiting the internal heat generation through the LIB's internal impedance. Fast external heating generates large temperature gradients across the LIB due to the low thermal conductivity of the cell, while internal impedance heating (usually through AC or pulse charge/discharging) tends to be relatively slow, although it can achieve more uniform temperature distribution. This paper investigates the potential of combining externally sourced resistive film heating with bidirectional pulse heating to achieve fast preheating without causing steep temperature gradients. The LIB is modeled with the Doyle Fuller Newman (DFN) electrochemical model and 1D thermal model, and reinforcement learning (RL) is used to optimize the pulse current amplitude and film voltage concurrently. The results indicate that the optimal policy for maximizing the rate of temperature rise while limiting temperature gradients has the film heating dominate the initial phases and create the ideal conditions for pulse heating to take over. In addition, the pulse component shares the heating load and reduces the energy rating of the auxiliary power source.

Emerging data-driven approaches, such as deep reinforcement learning (DRL), aim at on-the-field learning of powertrain control policies that optimize fuel economy and other performance metrics. Indeed, they have shown great potential in this regard for individual vehicles on specific routes or drive cycles. However, for fleets of vehicles that must service a distribution of routes, DRL approaches struggle with learning stability issues that result in high variances and challenge their practical deployment. In this paper, we present a novel framework for shared learning among a fleet of vehicles through the use of a distilled group policy as the knowledge sharing mechanism for the policy learning computations at each vehicle. We detail the mathematical formulation that makes this possible. Several scenarios are considered to analyze the functionality, performance, and computational scalability of the framework with fleet size. Comparisons of the cumulative performance of fleets using our proposed shared learning approach with a baseline of individual learning agents and another state-of-the-art approach with a centralized learner show clear advantages to our approach. For example, we find a fleet average asymptotic improvement of 8.5 percent in fuel economy compared to the baseline while also improving on the metrics of acceleration error and shifting frequency for fleets serving a distribution of suburban routes. Furthermore, we include demonstrative results that show how the framework reduces variance within a fleet and also how it helps individual agents adapt better to new routes.

Reinforcement learning (RL)-based driver assistance systems seek to improve fuel consumption via continual improvement of powertrain control actions considering experiential data from the field. However, the need to explore diverse experiences in order to learn optimal policies often limits the application of RL techniques in safety-critical systems like vehicle control. In this paper, an exponential control barrier function (ECBF) is derived and utilized to filter unsafe actions proposed by an RL-based driver assistance system. The RL agent freely explores and optimizes the performance objectives while unsafe actions are projected to the closest actions in the safe domain. The reward is structured so that driver's acceleration requests are met in a manner that boosts fuel economy and doesn't compromise comfort. The optimal gear and traction torque control actions that maximize the cumulative reward are computed via the Maximum a Posteriori Policy Optimization (MPO) algorithm configured for a hybrid action space. The proposed safe-RL scheme is trained and evaluated in car following scenarios where it is shown that it effectively avoids collision both during training and evaluation while delivering on the expected fuel economy improvements for the driver assistance system.

Reinforcement learning (RL)-based adaptive cruise control systems (ACC) that learn and adapt to road, traffic and vehicle conditions are attractive for enhancing vehicle energy efficiency and traffic flow. However, the application of RL in safety critical systems such as ACC requires strong safety guarantees which are difficult to achieve with learning agents that have a fundamental need to explore. In this paper, we derive control barrier functions as safety filters that allow an RL-based ACC controller to explore freely within a collision safe set. Specifically, we derive control barrier functions for high relative degree nonlinear systems to take into account inertia effects relevant to commercial vehicles. We also outline an algorithm for accommodating actuation saturation with these barrier functions. While any RL algorithm can be used as the performance ACC controller together with these filters, we implement the Maximum A Posteriori Policy Optimization (MPO) algorithm with a hybrid action space that learns fuel optimal gear selection and torque control policies. The safety filtering RL approach is contrasted with a reward shaping RL approach that only learns to avoid collisions after sufficient training. Evaluations on different drive cycles demonstrate significant improvements in fuel economy with the proposed approach compared to baseline ACC algorithms.

In order for automated mobile vehicles to navigate in the real world with minimal collision risks, it is necessary for their planning algorithms to consider uncertainties from measurements and environmental disturbances. In this paper, we consider analytical solutions for a conservative approximation of the mutual probability of collision between two robotic vehicles in the presence of such uncertainties. Therein, we present two methods, which we call unitary scaling and principal axes rotation, for decoupling the bivariate integral required for efficient approximation of the probability of collision between two vehicles including orientation effects. We compare the conservatism of these methods analytically and numerically. By closing a control loop through a model predictive guidance scheme, we observe through Monte-Carlo simulations that directly implementing collision avoidance constraints from the conservative approximations remains infeasible for real-time planning. We then propose and implement a convexification approach based on the tightened collision constraints that significantly improves the computational efficiency and robustness of the predictive guidance scheme.

Connected and automated vehicles (CAVs) have the potential to improve traffic throughput and achieve a more efficient utilization of the available roadway infrastructure. They also have the potential to reduce energy consumption through traffic motion harmonization, even when operating in mixed traffic with other human-driven vehicles. The key to realizing these potentials are coordinated control schemes that can be implemented in a distributed manner with the CAVs. In this paper, we propose a distributed predictive control framework that features a two-dimensional maneuver planner incorporating explicit coordination constraints between connected vehicles operating in mixed traffic at various penetration levels. The framework includes a distributed implementation of a reference speed assigner that estimates local traffic speed from on-board measurements and communicated information. We present an extensive evaluation of the proposed framework in traffic micro-simulations at various CAV penetrations from traffic flow, energy use, and lane utilization points of view. Results are compared to a baseline scenario with no CAVs, as well as, a benchmark one-dimensional planner.

Eco-driving strategies have been shown to provide significant reductions in fuel consumption. This paper outlines an active driver assistance approach that uses a residual policy learning (RPL) agent trained to provide residual actions to default power train controllers while balancing fuel consumption against other driver-accommodation objectives. Using previous experiences, our RPL agent learns improved traction torque and gear shifting residual policies to adapt the operation of the powertrain to variations and uncertainties in the environment. For comparison, we consider a traditional reinforcement learning (RL) agent trained from scratch. Both agents employ the off-policy Maximum A Posteriori Policy Optimization algorithm with an actor-critic architecture. By implementing on a simulated commercial vehicle in various car-following scenarios, we find that the RPL agent quickly learns significantly improved policies compared to a baseline source policy but in some measures not as good as those eventually possible with the RL agent trained from scratch., Comment: 10th IFAC Symposium on Advances in Automotive Control AAC 2022

With the growing need to reduce energy consumption and greenhouse gas emissions, Eco-driving strategies provide a significant opportunity for additional fuel savings on top of other technological solutions being pursued in the transportation sector. In this paper, a model-free deep reinforcement learning (RL) control agent is proposed for active Eco-driving assistance that trades-off fuel consumption against other driver-accommodation objectives, and learns optimal traction torque and transmission shifting policies from experience. The training scheme for the proposed RL agent uses an off-policy actor-critic architecture that iteratively does policy evaluation with a multi-step return and policy improvement with the maximum posteriori policy optimization algorithm for hybrid action spaces. The proposed Eco-driving RL agent is implemented on a commercial vehicle in car following traffic. It shows superior performance in minimizing fuel consumption compared to a baseline controller that has full knowledge of fuel-efficiency tables.

This paper experimentally demonstrates the effectiveness of an anticipative car-following algorithm in reducing energy use of gasoline engine and electric Connected and Automated Vehicles (CAV), without sacrificing safety and traffic flow. We propose a Vehicle-in-the-Loop (VIL) testing environment in which experimental CAVs driven on a track interact with surrounding virtual traffic in real-time. We explore the energy savings when following city and highway drive cycles, as well as in emergent highway traffic created from microsimulations. Model predictive control handles high level velocity planning and benefits from communicated intentions of a preceding CAV or estimated probable motion of a preceding human driven vehicle. A combination of classical feedback control and data-driven nonlinear feedforward control of pedals achieve acceleration tracking at the low level. The controllers are implemented in ROS and energy is measured via calibrated OBD-II readings. We report up to 30% improved energy economy compared to realistically calibrated human driver car-following without sacrificing following headway., Comment: Preprint Submission to Transportation Research Part C, 27 Pages, 20 Figures