Path Planning meets Machine Learning Ryo Yonetani | OMRON SINIC X Invited talk at International Workshop on Symbolic-Neural Learning (SNL) 2022 July 8, 2022 7/8/2022 SNL 2022 | (c) 2022 OMRON SINIC X. All Rights Reserved. 1 

Path Planning Single-agent path planning • Find a lowest-cost path for a single agent from its start to goal • While avoiding collisions with obstacles Multi-agent path planning • Find paths for a team of agents with e.g., lowest possible sum of costs • While avoiding collisions with obstacles and other agents 7/8/2022 SNL 2022 | (c) 2022 OMRON SINIC X. All Rights Reserved. 2 

Path Planning | How? This action is clearly bad as it collides with obstacles. This action is apparently good as it brings us closer to the goal. But there's a dead end ahead. This action should be the best as it brings us to the goal while avoiding dead ends. How can we get AI to think this way? Start Goal 7/8/2022 SNL 2022 | (c) 2022 OMRON SINIC X. All Rights Reserved. 3 

Path Planning | How? Classical algorithm? • Has long been studied • Design how to plan in a top-down fashion Machine learning? • Has greatly advanced recently • Learn how to plan in a bottom-up fashion 7/8/2022 SNL 2022 | (c) 2022 OMRON SINIC X. All Rights Reserved. 4 

Path Planning meets Machine Learning • Leveraging path planning AND machine learning • Key idea: Learning representations of problem instances that are effective for path planning algorithms 7/8/2022 SNL 2022 | (c) 2022 OMRON SINIC X. All Rights Reserved. 5 Neural network Transforms the representation of problem instance Problem instance Path planning algorithm Solves the transformed problem instance Path 

Path Planning meets Machine Learning • What can be improved by incorporating machine learning? • Performance: Improving planning optimality-efficiency trade-off (i.e., reduce planning cost to find shorter paths) • Generalizability: Enabling planning on raw-image inputs with unknown costs 7/8/2022 SNL 2022 | (c) 2022 OMRON SINIC X. All Rights Reserved. 6 ut A* Neural A* ut Human trajectory Neural A* Goal Input A* Neural A* Input Human trajectory Neural A* Goal Faster & near-optimal (a) Input A* Neura (b) Input Human trajectory Neura Start Goal (a) Input A* N (b) Input Human trajectory N Start Goal Planning without knowing actual costs 

Path Planning using Neural A* Search (ICML 2021) Ryo Yonetani1* Tatsunori Taniai1* *Equal contribution Mohammadamin1 Barekatain Mai Nishimura1 Asako Kanezaki2 1OMRON SINIC X 2Tokyo Institute of Technology 7/8/2022 SNL 2022 | (c) 2022 OMRON SINIC X. All Rights Reserved. 7 

A* Search • A*: Incrementally exploring the map from start to goal • Completeness: Able to find a solution path if one exists • Optimality: Able to find a shortest path • With machine learning, can we improve the performance and generalizability of A* in a principled fashion? 7/8/2022 SNL 2022 | (c) 2022 OMRON SINIC X. All Rights Reserved. 8 

Neural A* SNL 2022 | (c) 2022 OMRON SINIC X. All Rights Reserved. Backpropagation Trainable encoder Guidance map Differentiable A* Search result Problem instance 7/8/2022 9 

Neural A* | How It Works SNL 2022 | (c) 2022 OMRON SINIC X. All Rights Reserved. Performance: augmenting given cost maps to plan faster Generalizability: estimating cost maps from raw-image inputs to imitate ground-truth paths (a) Input A* Neural A* (b) Input Human trajectory Neural A* Start Goal (a) Input A* Neural A* (b) Input Human trajectory Neural A* Start Goal Guidance map = augmented cost map Guidance map = estimated cost map 7/8/2022 10 

Differentiable A* Making A*’s node selection and expansion steps differentiable… 7/8/2022 SNL 2022 | (c) 2022 OMRON SINIC X. All Rights Reserved. 11 G G Select Expand Node selection • Finding nodes from candidates for constructing a shortest path • Softmax + discretized activation Node expansion • Adding neighboring nodes to the list of next selection candidates • Fixed convolution + binary masking Select … G Candidates History 

Differentiable A* Learning to match differentiable A*’s search history with ground-truth path 7/8/2022 SNL 2022 | (c) 2022 OMRON SINIC X. All Rights Reserved. 12 G G Select Search history Ground-truth path Loss Loss: L1 between search history and ground-truth path • Backpropagated thorugh every search step to the encoder • Making guidance maps to indicate nodes to/not to explore 

Results | Shortest Path Search SNL 2022 | (c) 2022 OMRON SINIC X. All Rights Reserved. A* BF SAIL SAIL-SL BB-A* Neural BF Neural A* WA* Guidance map S G S G G S BF SAIL SAIL-SL BB-A* Neural BF Neural A* WA* Guidance map SAIL SAIL-SL BB-A* Neural BF Neural A* WA* Guidance map SAIL SAIL-SL BB-A* Neural BF Neural A* Guidance map (a) Search optimality (b) Search efficiency Hmean of (a) and (b) SAIL [Choudhury+, 2018] 34.6 48.6 26.3 BB-A* [Vlastelica+, 2020] 62.7 42.0 42.1 Neural A* 87.7 40.1 52.0 * The higher the better 7/8/2022 13 

Results | Planning to Imitate Pedestrian Trajectories SNL 2022 | (c) 2022 OMRON SINIC X. All Rights Reserved. Input Ground-truth BB-A* Neural A* Guidance map Task setup • Surveillance images + pedestrian trajectories as demonstrations • Predicting realistic trajectories consistent with those of pedestrians when start and goal locations are provided 7/8/2022 14 Chamfer distance (lower the better) BB-A* [Vlastelica+, 2020] 152.2 Neural A* 16.1 

CTRMs: Learning to Construct Cooperative Timed Roadmaps for Multi-agent Path Planning in Continuous Spaces (AAMAS 2022) Keisuke Okumura1,2 Ryo Yonetani1 Mai Nishimura1 Asako Kanezaki2 1OMRON SINIC X 2Tokyo Institute of Technology 7/8/2022 SNL 2022 | (c) 2022 OMRON SINIC X. All Rights Reserved. 15 

Multi-agent Path Planning (MAPP) 7/8/2022 SNL 2022 | (c) 2022 OMRON SINIC X. All Rights Reserved. 16 MAPP on the grid map (aka MAPF) • Limit agent locations and possible actions • Has long been studied for decades MAPP in continuous spaces (our focus) • Need to construct a graph (roadmap) that approximates the space • Has long been recognized as challenging and still relatively unexplored 

Roadmap Density vs Solution Quality/Efficiency 7/8/2022 SNL 2022 | (c) 2022 OMRON SINIC X. All Rights Reserved. 17 Sparser roadmaps • Can be constructed faster • May produce sub-optimal solutions or fail to solve the problem Denser roadmaps • Can produce higher-quality solutions • Require higher computational cost for construction and pathfinding 

Cooperative Timed Roadmaps (CTRM) • Cooperative: Roadmaps are constructed for each agent while being aware of other agents • Timed: Roadmaps are extended over the time axis to generate “timed” paths 7/8/2022 SNL 2022 | (c) 2022 OMRON SINIC X. All Rights Reserved. 18 Roadmap for agent #0 Roadmap for agent #1 Roadmap for agent #2 

Learning to Construct CTRMs • Key idea: Learning the distribution of possible next actions of each agent from the observations of others • Conditional VAE to learn the distribution • Attention mechanism to deal with variable number of agents 7/8/2022 SNL 2022 | (c) 2022 OMRON SINIC X. All Rights Reserved. 19 … Attention MLP MLP MLP Features for the self Features from other agents 

Constructing CTRMs to Solve MAPP • Constructing CTRMs: Iteratively sampling next actions using the learned CVAE and random sampler • Solving MAPP: Performing any multi-agent pathfinding algorithm on the constructed CTRMs 7/8/2022 SNL 2022 | (c) 2022 OMRON SINIC X. All Rights Reserved. 20 Constructing CTRMs New problem instance CTRMs Solution Solve MAPP 

Results 7/8/2022 SNL 2022 | (c) 2022 OMRON SINIC X. All Rights Reserved. 21 Compared to grid maps / PRMs, our approach • Achieves comparable solutions • Runs faster 

Summary • Path planning meets machine learning: = Learning representations of problem instances that are effective for path planning algorithms • Code available on GitHub • Neural A* | https://github.com/omron-sinicx/neural-astar • CTRM | https://github.com/omron-sinicx/jaxmapp 7/8/2022 SNL 2022 | (c) 2022 OMRON SINIC X. All Rights Reserved. 22 A* Neural A*