1. Tracking and Relative Localization of Drone Swarms With a Vision-Based Headset
- Author
-
Christopher Reardon, Fabrizio Schiano, Dario Floreano, Maxim Pavliv, Giuseppe Loianno, Pavliv, M., Schiano, F., Reardon, C. M., Floreano, D., and Loianno, G.
- Subjects
0209 industrial biotechnology ,Control and Optimization ,business.product_category ,Computer science ,Headset ,Biomedical Engineering ,02 engineering and technology ,Joint Probabilistic Data Association Filter ,Motion capture ,localization ,020901 industrial engineering & automation ,Artificial Intelligence ,Inertial measurement unit ,0202 electrical engineering, electronic engineering, information engineering ,Computer vision ,Headphones ,Robot kinematics ,Artificial neural network ,business.industry ,Mechanical Engineering ,Aerial Systems: Application ,Drone ,Computer Science Applications ,Human-Centered Robotic ,Human-Computer Interaction ,Control and Systems Engineering ,020201 artificial intelligence & image processing ,Computer Vision and Pattern Recognition ,Artificial intelligence ,human-centered robotics ,business ,Aerial systems, applications - Abstract
We address the detection, tracking, and relative localization of the agents of a drone swarm from a human perspective using a headset equipped with a single camera and an Inertial Measurement Unit (IMU). We train and deploy a deep neural network detector on image data to detect the drones. A joint probabilistic data association filter resolves the detection problems and couples this information with the headset IMU data to track the agents. In order to estimate the drones’ relative poses in 3D space with respect to the human, we use an additional deep neural network that processes image regions of the drones provided by the tracker. Finally, to speed up the deep neural networks’ training, we introduce an automated labeling process relying on a motion capture system. Several experimental results validate the effectiveness of the proposed approach. The approach is real-time, does not rely on any communication between the human and the drones, and can scale to a large number of agents, often called swarms. It can be used to spatially task a swarm of drones and also employed without a headset for formation control and coordination of terrestrial vehicles.