Search

This paper studies an unmanned aerial vehicle (UAV)-assisted wireless powered IoT network, where a rotary-wing UAV adopts fly-hover-communicate protocol to successively visit IoT devices in demand. During the hovering periods, the UAV works on full-duplex mode to simultaneously collect data from the target device and charge other devices within its coverage. Practical propulsion power consumption model and non-linear energy harvesting model are taken into account. We formulate a multi-objective optimization problem to jointly optimize three objectives: maximization of sum data rate, maximization of total harvested energy and minimization of UAV’s energy consumption over a particular mission period. These three objectives are in conflict with each other partly and weight parameters are given to describe associated importance. Since IoT devices keep gathering information from the physical surrounding environment and their requirements to upload data change dynamically, online path planning of the UAV is required. In this paper, we apply deep reinforcement learning algorithm to achieve online decision. An extended deep deterministic policy gradient (DDPG) algorithm is proposed to learn control policies of UAV over multiple objectives. While training, the agent learns to produce optimal policies under given weights conditions on the basis of achieving timely data collection according to the requirement priority and avoiding devices’ data overflow. The verification results show that the proposed MODDPG (multi-objective DDPG) algorithm achieves joint optimization of three objectives and optimal policies can be adjusted according to weight parameters among optimization objectives. [ABSTRACT FROM AUTHOR]

With the proliferation of the Internet of Things and smart devices, there exists an urge to address the critical computation demands of end users for several real-time applications. Fog computing (FC) targets to address the computation requirements nearer to the end user in almost real-time. Mainly, the fog environments (FE) exhibit characteristics of resource constraint and wide heterogeneity. In addition, the complex fog environment enforces the fog resources (FR) deployment either within fixed or distributed over random geographical locations. In the literature, there exist several solutions for the optimization of FR allocation. However, the location-dependent FR allocation has not been addressed thoroughly. Hence, this paper proposes a novel hybrid clustering-based deep Q-network for location-based optimal resource allocation (HCDQN-ORA) model. Here, the first objective is to minimize the search operation for identifying suitable FR and to meet the quality of service (QoS) demand using an enhanced fog resource clustering (E-FRC) algorithm. The second objective aims to achieve optimal FR allocation among the set of location-dependent FRs using reinforcement learning techniques. Problem formulation has been modelled using Markov decision process. A deep Q-network algorithm with two variants, namely enhanced optimal resource allocation using deep Q-network (EORA-DQN) and enhanced optimal resource allocation using deep Q-network with experience replay (EORA-DQN-ER), has been incorporated. To analyse the statically located and randomly distributed FR environment, the experimental results exhibit that, in the case of clustering of FRs, E-FRC is more efficient than conventional clustering algorithms. Next, the location-dependent optimal FR allocation problem incorporates the performance analysis based on the convergence of loss function and the parametric analysis based on batch size, respectively. Results exhibit that EORA-DQN performs better only for statically allotted resources. However, the proposed EORA-DQN-ER outperforms both types of FR with a success rate of 89.6% of optimal allocation with the batch size of 200. [ABSTRACT FROM AUTHOR]

With increasing growth in IoT, the number of devices connected to the Internet is constantly growing. Moreover, the increase in the volume of data and their transmission through the Internet of Things, as well as the existence of inadequate bandwidth, limits cloud-based storage and data processing. Both fog and cloud computing provide the storage space, application, and data for users; however, fog is more proximate to the end user with wider geographical distribution. When bringing the computing resources closer to the required location in the fog environment, the efficiency of the system increases, and the distance at which data must be transmitted decreases. On the other hand, implementing IoT applications and satisfying the requests of end users in fog computing will create new challenges in resource allocation and dynamic resource provisioning. The flexible and usually automatic mechanisms require the determination of required virtual resources to minimize the resource consumption and service level agreement (SLA). In this paper, we introduce a framework for increasing resource management efficiency in the IoT ecosystem based on deep reinforcement learning (DRL). The proposed deep neural network (DNN) method for estimating value functions improves adaptability to different oscillating conditions, learns past sensible strategies, and as a self-learning adaptive system by replicating interactions with the fog environment. The DRL algorithm finds the best destination for implementing IoT services to compromise between minimizing average power consumption, minimizing average service latency, reducing costs, and balancing resource allocation. Finally, through simulations, we show that under different loading rates, the policy used compared to other comparable solutions is to increase utilization and reduce the rate of delay, while ensuring an acceptable level of service quality. [ABSTRACT FROM AUTHOR]

The Routing Protocol for Low power and lossy networks (RPL) is used as a routing protocol in IoT applications. In an endeavor to bring out an optimized approach for providing Quality of Service (QoS) routing for heavy volume IoT data transmissions this paper proposes a machine learning-based routing algorithm with a multi-agent environment. The overall routing process is divided into two phases: route discovery phase and route maintenance phase. The route discovery or path finding phase is performed using rank calculation and Q-routing. Q-routing is performed with Q-Learning reinforcement machine learning approach, for selecting the next hop node. The proposed routing protocol first creates a Destination Oriented Directed Acyclic Graph (DODAG) using Q-Learning. The second phase is route maintenance. In this paper, we also propose an approach for route maintenance that considerably reduces control overheads as shown by the simulation and has shown less delay in routing convergence. [ABSTRACT FROM AUTHOR]

The integration of new Internet of Things (IoT) applications and services heavily relies on task offloading to external devices due to the constrained computing and battery resources of IoT devices. Up to now, Cloud Computing (CC) paradigm has been a good approach for tasks where latency is not critical, but it is not useful when latency matters, so Multi-access Edge Computing (MEC) can be of use. In this work, we propose a distributed Deep Reinforcement Learning (DRL) tool to optimize the binary task offloading decision, this is, the independent decision of where to execute each computing task, depending on many factors. The optimization goal in this work is to maximize the Quality-of-Experience (QoE) when performing tasks, which is defined as a metric related to the battery level of the UE, but subject to satisfying tasks' latency requirements. This distributed DRL approach, specifically an Actor-Critic (AC) algorithm running on each User Equipment (UE), is evaluated through the simulation of two distinct scenarios and outperforms other analyzed baselines in terms of QoE values and/or energy consumption in dynamic environments, also demonstrating that decisions need to be adapted to the environment's evolution. [ABSTRACT FROM AUTHOR]

The Internet of things and medical things (IoT) and (IoMT) technologies have been deployed to simplify humanity's life, which the complexity of communications between their layers was increased by rising joining the applications to IoT and IoMT-based infrastructures. The issue is challenging for decision-making and the quality of service where some researchers addressed the reward-based methods to tackle the problems by employing reinforcement learning (RL) algorithms and deep neural networks (DNNs). Nevertheless, satisfying its availability remains a challenge for the quality of service due to the lack of imposing a penalty to the defective devices after detecting faults. This paper proposes a quasi-mapping method to transfer the roles of sensors and services onto a neural network's nodes to satisfy IoT-based applications' availability using a penalty-backwarding approach into the NN's weights and prunes weak neurons and synaptic weights (SWs). We reward the sensors and fog services, and the connection weights between them when are covered the defective nodes' output. Additionally, this work provides a decision-making approach to dedicate the suitable service to the requester using employing a threshold value in the NN's output layer according to the application. By providing an intelligent algorithm, the study decides to provide a service based on its availability and updating initial information, including faulty devices and new joined components. The observations and results prove decision-making accuracy for different IoT-based applications by approximately 95.8–97% without imposing the cost. The study reduces energy consumption and delay by approximately 64.71% and 47.4% compared without using neural networks besides creating service availability. This idea affects deploying IoT infrastructures to decision-making about providing appropriate services in critical situations because of removing defective devices and joining new components by imposing penalties and rewards by the designer, respectively. [ABSTRACT FROM AUTHOR]

Unmanned aerial vehicle (UAV) plays a more and more important role in Internet of Things (IoT) for remote sensing and device interconnecting. Due to the limitation of computing capacity and energy, the UAV cannot handle complex tasks. Recently, computation offloading provides a promising way for the UAV to handle complex tasks by deep reinforcement learning (DRL)-based methods. However, existing DRL-based computation offloading methods merely protect usage pattern privacy and location privacy. In this paper, we consider a new privacy issue in UAV-assisted IoT, namely computation offloading preference leakage, which lacks through study. To cope with this issue, we propose a novel privacy-preserving online computation offloading method for UAV-assisted IoT. Our method integrates the differential privacy mechanism into deep reinforcement learning (DRL), which can protect UAV's offloading preference. We provide the formal analysis on security and utility loss of our method. Extensive real-world experiments are conducted. Results demonstrate that, compared with baseline methods, our method can learn cost-efficient computation offloading policy without preference leakage and a priori knowledge of the wireless channel model. [ABSTRACT FROM AUTHOR]

The internet of things (IoT) represents a disruptive concept that has been changing society in several ways. There have been several successful applications of IoT in the industry. For example, in transportation systems, the novel internet of vehicles (IoV) concept has enabled new research directions and automation solutions. Moreover, reinforcement learning (RL), federated learning (FL), and federated reinforcement learning (FRL) have demonstrated remarkable success in solving complex problems in different applications. In recent years, new solutions have been developed based on this combined framework (i.e., federated reinforcement learning). Conversely, there is a lack of analysis concerning IoT applications and a standard view of challenges and future directions of the current FRL landscape. Thereupon, the main goal of this research is to present a literature review of federated reinforcement learning (FRL) applications in IoT from multiple perspectives. We focus on analyzing applications in multiple areas (e.g., security, sustainability and efficiency, vehicular solutions, and industrial services) to highlight existing solutions, their characteristics, and research gaps. Additionally, we identify key short- and long-term challenges leading to new opportunities in the field. This research intends to picture the current FRL ecosystem in IoT to foster the development of new solutions based on existing challenges. [ABSTRACT FROM AUTHOR]

Edge computing (EC) has recently emerged as a promising paradigm that supports resource-hungry Internet of Things (IoT) applications with low latency services at the network edge. However, the limited capacity of computing resources at the edge server poses great challenges for scheduling application tasks. In this paper, a task scheduling problem is studied in the EC scenario, and multiple tasks are scheduled to virtual machines (VMs) configured at the edge server by maximizing the long-term task satisfaction degree (LTSD). The problem is formulated as a Markov decision process (MDP) for which the state, action, state transition, and reward are designed. We leverage deep reinforcement learning (DRL) to solve both time scheduling (i.e., the task execution order) and resource allocation (i.e., which VM the task is assigned to), considering the diversity of the tasks and the heterogeneity of available resources. A policy-based REINFORCE algorithm is proposed for the task scheduling problem, and a fully-connected neural network (FCN) is utilized to extract the features. Simulation results show that the proposed DRL-based task scheduling algorithm outperforms the existing methods in the literature in terms of the average task satisfaction degree and success ratio. [ABSTRACT FROM AUTHOR]

In the Internet of Things (IoT), Low-Power Wide-Area Networks (LPWANs) are designed to provide low energy consumption while maintaining a long communications' range for End Devices (EDs). LoRa is a communication protocol that can cover a wide range with low energy consumption. To evaluate the efficiency of the LoRa Wide-Area Network (LoRaWAN), three criteria can be considered, namely, the Packet Delivery Rate (PDR), Energy Consumption (EC), and coverage area. A set of transmission parameters have to be configured to establish a communication link. These parameters can affect the data rate, noise resistance, receiver sensitivity, and EC. The Adaptive Data Rate (ADR) algorithm is a mechanism to configure the transmission parameters of EDs aiming to improve the PDR. Therefore, we introduce a new algorithm using the Multi-Armed Bandit (MAB) technique, to configure the EDs' transmission parameters in a centralized manner on the Network Server (NS) side, while improving the EC, too. The performance of the proposed algorithm, the Low-Power Multi-Armed Bandit (LP-MAB), is evaluated through simulation results and is compared with other approaches in different scenarios. The simulation results indicate that the LP-MAB's EC outperforms other algorithms while maintaining a relatively high PDR in various circumstances. [ABSTRACT FROM AUTHOR]

With the advent of the Internet of Things (IoT), the security of the network layer in IoT is getting more attention in recent decades. Various intrusion detection methods were developed in the existing research works, but the capability to detect malicious and intrusion activities in the complex Internet environment poses a challenging task in IoT. Hence, an effective and optimal intrusion detection mechanism, named Harmony Search Hawks Optimization-based Deep Reinforcement Learning (HSHO-based Deep RL), is proposed in this research to detect malicious network activities. The proposed Harmony Search Hawks Optimization (HSHO) algorithm is designed by integrating Harmony Search (HS) with the Harris Hawks Optimization (HHO) algorithm. However, the optimal detection result that is effectively achieved through the fitness measures such that the minimum fitness value is only declared as the optimal solution. The Deep Reinforcement Learning (Deep RL) classifier effectively detects the malicious or intruder behaviors and generates a satisfactory result. By reducing the dimensionality of data using nonnegative matrix factorization, the data is optimally fit to perform intrusion detection process in the IoT environment. The proposed HSHO-based Deep RL obtained better performance in terms of the metrics like accuracy (96.925%), True Positive Rate (TPR; 96.90%), and True Negative Rate (TNR; 97.920%) with respect to K-fold. [ABSTRACT FROM AUTHOR]

Low power wide area networks (LPWANs) have been recently deployed for long-range machine-to-machine (M2M) communications. These networks have been proposed for many applications and in particular for the communications of the advanced metering infrastructure (AMI) backhaul of the smart grid. However, they rely on simple access schemes that may suffer from important latency, which is one of the main performance indicators in smart grid communications. In this article, we apply reinforcement learning (RL) algorithms to reduce the latency of AMI communications in LPWANs. For that purpose, we first study the collision probability in an unslotted ALOHA-based LPWAN AMI backhaul which uses the LoRaWAN acknowledgement procedure. Then, we analyse the effect of collisions on the latency for different frequency access schemes. We finally show that RL algorithms can be used for the purpose of frequency selection in these networks and reduce the latency of the AMI backhaul in LPWANs. Numerical results show that non-coordinated algorithms featuring a very low complexity reduce the collision probability by 14% and the mean latency by 40%. [ABSTRACT FROM AUTHOR]