Multi-Objective MDP-Based Routing in UAV Networks for Search-Based Operations

Unmanned aerial vehicle (UAV) systems have gained widespread recognition due to their versatility and autonomy. Their deployment for disaster mitigation and management operations is seen as one of their most important applications over the past decade. In such UAV networks, routing plays a crucial role in determining network performance parameters such as network lifetime, data transmission latency, and packet delivery ratio. This paper presents a novel routing mechanism - Multi-Objective Markov Decision Based Routing (MOBMDP) for UAV networks carrying out search-based operations. MOBMDP models routing decisions in a Markov Decision Process (MDP) framework and uses Q-learning to take decisions. It compares routing paths using three metrics, viz., Remaining Energy of the Minimum Energy Node (REMEN), Power Distance ratio (PD), and Expected Delay (ED). Amongst these metrics, PD is a novel metric proposed by this work. PD simultaneously optimizes the energy efficiency and energy distribution in the network. Further, MOBMDP uses a novel reinforcement learning inspired method to estimate transmission delay in a given path. Intensive simulation studies compare MOBMDP to four state-of-the-art routing protocols. Results show a significant improvement in network lifetime, packet delivery ratio, energy efficiency, average data transmission delay, and error in delay estimation.