How To Evaluate the Navigation of Autonomous Vehicles Around Pedestrians?

The navigation of autonomous vehicles around pedestrians is a key challenge when driving in urban environments. It is essential to test the proposed navigation system using simulation before moving to real-life implementation and testing. Evaluating the performance of the system requires the design of a diverse set of tests which spans the targeted working scenarios and conditions. These tests can then undergo a process of evaluation using a set of adapted performance metrics. This work addresses the problem of performance evaluation for an autonomous vehicle in a shared space with pedestrians. The methodology for designing the test simulations is discussed. Moreover, a group of performance metrics is proposed to evaluate the different aspects of the navigation: the motion safety, the quality of the generated trajectory and the comfort of the pedestrians surrounding the vehicle. Furthermore, the success/fail criterion for each metric is discussed. The implementation of the proposed evaluation method is illustrated by evaluating the performance of a pre-designed proactive navigation system using a shared space crowd simulator under Robot Operating System (ROS).