Search

We present H-TD²: Hybrid Temporal Difference Learning for Taxi Dispatch, a model-free, adaptive decision-making algorithm to coordinate a large fleet of automated taxis in a dynamic urban environment to minimize expected customer waiting times. Our scalable algorithm exploits the natural transportation network company topology by switching between two behaviors: distributed temporal-difference learning computed locally at each taxi and infrequent centralized Bellman updates computed at the dispatch center. We derive a regret bound and design the trigger condition between the two behaviors to explicitly control the trade-off between computational complexity and the individual taxi policy's bounded sub-optimality; this advances the state of the art by enabling distributed operation with bounded-suboptimality. Additionally, unlike recent reinforcement learning dispatch methods, this policy estimation is adaptive and robust to out-of-training domain events. This result is enabled by a two-step modelling approach: the policy is learned on an agent-agnostic, cell-based Markov Decision Process and individual taxis are coordinated using the learned policy in a distributed game-theoretic task assignment. We validate our algorithm against a receding horizon control baseline in a Gridworld environment with a simulated customer dataset, where the proposed solution decreases average customer waiting time by 50% over a wide range of parameters. We also validate in a Chicago city environment with real customer requests from the Chicago taxi public dataset where the proposed solution decreases average customer waiting time by 26% over irregular customer distributions during a 2016 Major League Baseball World Series game.

We present a self-improving, Neural Tree Expansion (NTE) method for multi-robot online planning in non-cooperative environments, where each robot attempts to maximize its cumulative reward while interacting with other self-interested robots. Our algorithm adapts the centralized, perfect information, discrete-action space method from AlphaZero to a decentralized, partial information, continuous action space setting for multi-robot applications. Our method has three interacting components: (i) a centralized, perfect-information "expert" Monte Carlo Tree Search (MCTS) with large computation resources that provides expert demonstrations, (ii) a decentralized, partial-information "learner" MCTS with small computation resources that runs in real-time and provides self-play examples, and (iii) policy & value neural networks that are trained with the expert demonstrations and bias both the expert and the learner tree growth. Our numerical experiments demonstrate Neural Tree Expansion's computational advantage by finding better solutions than a MCTS with 20 times more resources. The resulting policies are dynamically sophisticated, demonstrate coordination between robots, and play the Reach-Target-Avoid differential game significantly better than the state-of-the-art control-theoretic baseline for multi-robot, double-integrator systems. Our hardware experiments on an aerial swarm demonstrate the computational advantage of Neural Tree Expansion, enabling online planning at 20Hz with effective policies in complex scenarios., Accepted at IEEE RA-L, see DOI below

Microvascular invasion (MVI) is one of the main prognostic factors of hepatocellular carcinoma (HCC) after liver transplantation (LT), but its occurrence is unpredictable before surgery. The alpha fetoprotein (AFP) model (composite score including size, number, AFP), currently used in France, defines the selection criteria for LT. This study’s aim was to evaluate the preoperative predictive value of AFP SCORE progression on MVI and overall survival during the waiting period for LT. Data regarding LT recipients for HCC from 2007 to 2015 were retrospectively collected from a single institutional database. Among 159 collected cases, 34 patients progressed according to AFP SCORE from diagnosis until LT. MVI was shown to be an independent histopathological prognostic factor according to Cox regression and competing risk analysis in our cohort. AFP SCORE progression was the only preoperative predictive factor of MVI (OR = 10.79 [2.35–49.4]; p 0.002). The 5-year overall survival in the progression and no progression groups was 63.9% vs. 86.3%, respectively (p = 0.001). Cumulative incidence of HCC recurrence was significantly different between the progression and no progression groups (Sub-HR = 4.89 [CI 2–11.98]). In selected patients, the progression of AFP SCORE during the waiting period can be a useful preoperative tool to predict MVI.

We present GLAS: Global-to-Local Autonomy Synthesis, a provably-safe, automated distributed policy generation for multi-robot motion planning. Our approach combines the advantage of centralized planning of avoiding local minima with the advantage of decentralized controllers of scalability and distributed computation. In particular, our synthesized policies only require relative state information of nearby neighbors and obstacles, and compute a provably-safe action. Our approach has three major components: i) we generate demonstration trajectories using a global planner and extract local observations from them, ii) we use deep imitation learning to learn a decentralized policy that can run efficiently online, and iii) we introduce a novel differentiable safety module to ensure collision-free operation, thereby allowing for end-to-end policy training. Our numerical experiments demonstrate that our policies have a 20% higher success rate than optimal reciprocal collision avoidance, ORCA, across a wide range of robot and obstacle densities. We demonstrate our method on an aerial swarm, executing the policy on low-end microcontrollers in real-time., Accepted at IEEE RA-L, see DOI below

In this paper, a learning-based optimal transportation algorithm for autonomous taxis and ridesharing vehicles is introduced. The goal is to design a mechanism to solve the routing problem for a fleet of autonomous vehicles in real-time in order to maximize the transportation company’s profit. To solve this problem, the system is modeled as a Markov Decision Process (MDP) using past customers data. By solving the defined MDP, a centralized high-level planning recommendation is obtained, where this offline solution is used as an initial value for the real-time learning. Then, a distributed SARSA reinforcement learning algorithm is proposed to capture the model errors and the environment changes, such as variations in customer distributions in each area, traffic, and fares, thereby providing an accurate model and optimal policies in real-time. Agents are using only their local information and interaction, such as current passenger requests and estimates of neighbors’ tasks and their optimal actions, to obtain the optimal policies in a distributed fashion. The agents use the estimated values of each action, provided by distributed SARSA reinforcement learning, in a distributed game-theory based task assignment to select their conflict-free customers. Finally, the customers data provided by the city of Chicago is used to validate the proposed algorithms.