Search

This paper proposes a novel scalable reinforcement learning approach for simultaneous routing and spectrum access in wireless ad-hoc networks. In most previous works on reinforcement learning for network optimization, the network topology is assumed to be fixed, and a different agent is trained for each transmission node—this limits scalability and generalizability. Further, routing and spectrum access are typically treated as separate tasks. Moreover, the optimization objective is usually a cumulative metric along the route, e.g., number of hops or delay. In this paper, we account for the physical-layer signal-to-interference-plus-noise ratio (SINR) in a wireless network and further show that bottleneck objective such as the minimum SINR along the route can also be optimized effectively using reinforcement learning. Specifically, we propose a scalable approach in which a single agent is associated with each flow and makes routing and spectrum access decisions as it moves along the frontier nodes. The agent is trained according to the physical-layer characteristics of the environment using a novel rewarding scheme based on the Monte Carlo estimation of the future bottleneck SINR. It learns to avoid interference by intelligently making joint routing and spectrum allocation decisions based on the geographical location information of the neighbouring nodes. [ABSTRACT FROM AUTHOR]

This paper applies machine learning to optimize the transmission policies of cognitive radio inspired non-orthogonal multiple access (CR-NOMA) networks, where time-division multiple access (TDMA) is used to serve multiple primary users and an energy-constrained secondary user is admitted to the primary users’ time slots via NOMA. During each time slot, the secondary user performs the two tasks: data transmission and energy harvesting based on the signals received from the primary users. The goal of the paper is to maximize the secondary user’s long-term throughput, by optimizing its transmit power and the time-sharing coefficient for its two tasks. The long-term throughput maximization problem is challenging due to the need for making decisions that yield long-term gains but might result in short-term losses. For example, when in a given time slot, a primary user with large channel gains transmits, intuition suggests that the secondary user should not carry out data transmission due to the strong interference from the primary user but perform energy harvesting only, which results in zero data rate for this time slot but yields potential long-term benefits. In this paper, a deep reinforcement learning (DRL) approach is applied to emulate this intuition, where the deep deterministic policy gradient (DDPG) algorithm is employed together with convex optimization. Our simulation results demonstrate that the proposed DRL assisted NOMA transmission scheme can yield significant performance gains over two benchmark schemes. [ABSTRACT FROM AUTHOR]

This paper proposes an effective and novel multi-agent deep reinforcement learning (MADRL)-based method for solving the joint virtual network function (VNF) placement and routing (P&R), where multiple service requests with differentiated demands are delivered at the same time. The differentiated demands of the service requests are reflected by their delay- and cost-sensitive factors. We first construct a VNF P&R problem to jointly minimize a weighted sum of service delay and resource consumption cost, which is NP-complete. Then, the joint VNF P&R problem is decoupled into two iterative subtasks: placement subtask and routing subtask. Each subtask consists of multiple concurrent parallel sequential decision processes. By invoking the deep deterministic policy gradient method and multi-agent technique, an MADRL-P&R framework is designed to perform the two subtasks. The new joint reward and internal rewards mechanism is proposed to match the goals and constraints of the placement and routing subtasks. We also propose the parameter migration-based model-retraining method to deal with changing network topologies. Corroborated by experiments, the proposed MADRL-P&R framework is superior to its alternatives in terms of service cost and delay, and offers higher flexibility for personalized service demands. The parameter migration-based model-retraining method can efficiently accelerate convergence under moderate network topology changes. [ABSTRACT FROM AUTHOR]

Wireless body area networks (WBANs) are vulnerable to active eavesdropping that simultaneously perform sniffing and jamming to raise the sensor transmit power, and thus steal more healthcare data. In this paper, we propose an intelligent reflecting surface (IRS)-aided reinforcement learning (RL) based secure WBAN transmission scheme that enables the coordinator to jointly optimize the sensor encryption key and transmit power, as well as the IRS phase shifts against active eavesdropping. A Dyna architecture is designed to improve the learning efficiency with the simulated transmission experiences and safe exploration is applied to avoid the risky policies that result in severe data leakage. A deep RL based WBAN transmission scheme is proposed to further improve the secure transmission with lower eavesdropping rate, intercept probability, sensor energy consumption and transmission latency for the coordinators that support deep learning. We analyze the computational complexity and investigate the equilibrium of the secure transmission game between the coordinator and the eavesdropper to provide the performance bounds, which is verified via the simulation results, showing the efficacy of our proposed schemes. [ABSTRACT FROM AUTHOR]

The renewable energy source (RES)-powered small cell base station (SBS) is a promising technology for the next-generation networks because RESs provide sustainable energy without being depleted. In RES-assisted networks, joint optimization of user association (UA) and power control (PC) is required to enhance the sum-rate performance by reducing inter-cell interference between SBSs. This paper tackles the UA and PC optimization for the sum-rate maximization under quality-of-service (QoS) and backhaul constraints. We formulate the problem as mixed-integer non-linear programming, in which UA and PC variables of different time slots are tightly coupled due to the RES energy dynamics model. Hence, designing a dynamic policy-based UA and PC with consideration of the future environments such as the quantity of channel gain and energy harvesting remains a challenge. First, we propose a deep unsupervised learning (DUL)-based UA scheme for a fixed PC variable. To lower the computational complexity and accelerate the convergence of the proposed learning-based optimization, we relax the UA variable, which originally has an extremely high dimension, into a low-dimensional continuous variable inspired by the Lagrangian method. Next, we propose a deep reinforcement learning (DRL)-based PC scheme, in which the stringent QoS and backhaul constraints are considered penalty terms on the reward design. The proposed DRL-based PC scheme facilitates dynamic PC inferring the relationship between the future environment and current PC. Simulation results demonstrate that the proposed scheme enhances the sum-rate by 10%, accommodates 3.3 percent point (%p) more QoS-qualified users, and reduces the computation time by 20 times compared to the conventional optimization-based method. [ABSTRACT FROM AUTHOR]

Many of the existing optimization-based transmit power control algorithms suffer from high computational complexity and require instantaneous global channel state information (CSI), both of which hinder their practical implementation. In this paper, we consider a wireless network with multiple transmitter-receiver pairs, where each transmitter only has access to its local CSI fed back from its intended receiver and does not require local CSI exchange with its neighboring transmitters. In such a network scenario, we propose a deep reinforcement learning based decentralized multi-agent power control (DEC-MAPC) algorithm for sum-rate maximization, where each transmitter acts as an intelligent agent. By leveraging the value decomposition technique, we establish a nonlinear mapping from the local reward of each agent to the global reward. Such a design allows each agent to independently control its transmit power based on its local CSI while enabling global collaboration among the agents. The proposed algorithm is scalable to large-scale networks as only local CSI is required, and is robust to the channel and interference variations via interacting with the environment. Simulation results show that the proposed DEC-MAPC algorithm with local CSI achieves comparable sum-rate performance with the centralized optimization algorithms with global CSI, while significantly reducing the computational complexity. [ABSTRACT FROM AUTHOR]

In this paper, we aim to propose a novel transmission control protocol (TCP) congestion control method from a cross-layer-based perspective and present a deep reinforcement learning (DRL)-driven method called DRL-3R (DRL for congestion control with Radio access network information and Reward Redistribution) so as to learn the TCP congestion control policy in a superior manner. In particular, we incorporate the RAN information to timely grasp the dynamics of RAN, and empower DRL to learn from the delayed RAN information feedback potentially induced by several consecutive actions. Meanwhile, we relax the implicit assumption (that the feedback to one specific action returns at a round-trip-time (RTT) after the action is applied) in previous researches, by redistributing the rewards and evaluating the merits of actions more accurately. Experiment results show that besides maintaining a reasonable fairness, DRL-3R significantly outperforms classical congestion control methods (e.g., TCP Reno, Westwood, Cubic, BBR and DRL-CC) on network utility by achieving a higher throughput while reducing delay in various network environments. [ABSTRACT FROM AUTHOR]

In this paper, we study the outage probability minimizing problem in a two-hop cooperative relay network. To reduce outage probability, existing studies propose many schemes for relay selection and power allocation, which are usually based on the assumption of exact channel state information (CSI). However, it is difficult to obtain perfect instantaneous CSI in practical situations where channel states change rapidly, and thus traditional methods would not perform well. Considering these factors, we turn to the emerging reinforcement learning (RL) methods for solutions. RL methods do not need any prior knowledge of CSI, but use neural network for approximation and decision after interacting with communication environment. Nevertheless, conventional RL methods, including most deep reinforcement learning (DRL) methods, cannot perform well when the search space is too large. In addition, non-stationarity is a common problem when using hierarchical reinforcement learning (HRL), which is caused by the changing behavior in different hierarchies. Therefore, we first propose a DRL framework with an outage-based reward function, which is then used as a baseline. Then, we further design an HRL framework and training algorithm. By decomposing relay selection and power allocation into two hierarchical optimization objectives, and combining on- policy and off-policy methods in the HRL framework, our method successfully address the sparse reward and non-stationary problem. Simulation results reveal that compared with traditional DRL method, the proposed HRL training algorithm can converge faster and reduce the outage probability by 8% in two-hop relay network with the same outage threshold. [ABSTRACT FROM AUTHOR]

In the multi-server multi-access edge computing (MEC) system, the computing capacity could be insufficient and the computing resource could be over-utilized when an excessive number of computing tasks are offloaded for execution. To alleviate the excessive burden in the multi-server MEC system and tap the underutilized resources of wired edge devices, which are called edge computers in this paper, we investigate the computation offloading and shunting scheme in the wireless wireline internetwork. We formulate an optimization problem to minimize the average total time delay and use the deep reinforcement learning method to obtain the optimal shunting policy. By sensing the state of the wireless wireline internetwork, the proposed scheme can adaptively adjust the shunting ratio and utilize the resources of MEC servers and edge computers intelligently and jointly. Finally, we evaluate the computational complexity and validate the convergence property of this scheme. The extensive simulation results demonstrate that this scheme can improve the resource productivity and reduce the average total time delay. [ABSTRACT FROM AUTHOR]

Network slicing is a key paradigm in 5G and is expected to be inherited in future 6G networks for the concurrent provisioning of diverse quality of service (QoS). Unfortunately, effective slicing of Radio Access Networks (RAN) is still challenging due to time-varying network situations. This paper proposes a new intelligent RAN slicing strategy with two-layered control granularity, which aims at maximizing both the long-term QoS of services and spectrum efficiency (SE) of slices. The proposed method consists of an upper-level controller to ensure the QoS performance, which enforces loose control by performing adaptive slice configuration according to the long-term dynamics of service traffic. The lower-level controller is to improve SE of slices, by tightly scheduling radio resources to users at the small time-scale. To realize the proposed RAN slicing strategy, we propose a model-free deep reinforcement learning (DRL) framework, which is a hierarchical structure that collaboratively integrating the modified deep deterministic policy gradient (DDPG) and double deep-Q-network algorithm. Specifically, the lower-level control problem is a mixed-integer stochastic optimization problem with multiple constraints. This kind of problem is hard to be directly solved by the exiting DRL algorithms, since it involves searching for the solution in a vast set of mixed-integer action space, which will induce unbearable computational complexity. Thus, we propose a novel action space reducing approach, embedding the convex optimization tools into the DDPG algorithm, to speed up the lower-level control. Furthermore, simulation results confirm the effectiveness of our proposed intelligent RAN slicing scheme. [ABSTRACT FROM AUTHOR]

Unmanned aerial vehicles (UAVs) that are widely utilized for video capturing, processing and transmission have to address jamming attacks with dynamic topology and limited energy. In this paper, we propose a reinforcement learning (RL)-based UAV anti-jamming video transmission scheme to choose the video compression quantization parameter, the channel coding rate, the modulation and power control strategies against jamming attacks. More specifically, this scheme applies RL to choose the UAV video compression and transmission policy based on the observed video task priority, the UAV-controller channel state and the received jamming power. This scheme enables the UAV to guarantee the video quality-of-experience (QoE) and reduce the energy consumption without relying on the jamming model or the video service model. A safe RL-based approach is further proposed, which uses deep learning to accelerate the UAV learning process and reduce the video transmission outage probability. The computational complexity is provided and the optimal utility of the UAV is derived and verified via simulations. Simulation results show that the proposed schemes significantly improve the video quality and reduce the transmission latency and energy consumption of the UAV compared with existing schemes. [ABSTRACT FROM AUTHOR]

In recent years, unmanned aerial vehicles (UAVs) have unlocked numerous sensing applications, which are expected to add billions of dollars to the world economy in the next decade. To further improve the Quality-of-Service in these applications, the 3rd Generation Partnership Project has considered the use of terrestrial cellular networks to support UAV sensing services, also known as the cellular Internet of UAVs. In this paper, we consider a cellular Internet of UAVs, where the sensory data can be transmitted either to the base station via cellular links, or to the mobile devices by underlay UAV-to-Device (U2D) communications. To evaluate the freshness of the sensory data, the concept of age of information (AoI) is adopted, in which a lower AoI implies fresher data. Since UAVs’ AoIs are determined by their trajectories during sensing and transmission, we investigate the AoI minimization problem for UAVs by designing their trajectories. This problem is a Markov decision problem with an infinite state-action space, and thus we utilize multi-agent deep reinforcement learning to approximate the state-action space. Then, we propose a multi-UAV trajectory design algorithm to solve this problem. Simulation results show that our proposed algorithm can achieve a lower AoI than a greedy algorithm, policy gradient algorithm, and overlay U2D scheme. [ABSTRACT FROM AUTHOR]

In this paper, a novel experienced deep reinforcement learning (deep-RL) framework is proposed to provide model-free resource allocation for ultra reliable low latency communication (URLLC-6G) in the downlink of a wireless network. The goal is to guarantee high end-to-end reliability and low end-to-end latency, under explicit data rate constraints, for each wireless user without any models of or assumptions on the users’ traffic. In particular, in order to enable the deep-RL framework to account for extreme network conditions and operate in highly reliable systems, a new approach based on generative adversarial networks (GANs) is proposed. This GAN approach is used to pre-train the deep-RL framework using a mix of real and synthetic data, thus creating an experienced deep-RL framework that has been exposed to a broad range of network conditions. The proposed deep-RL framework is particularly applied to a multi-user orthogonal frequency division multiple access (OFDMA) resource allocation system. Formally, this URLLC-6G resource allocation problem in OFDMA systems is posed as a power minimization problem under reliability, latency, and rate constraints. To solve this problem using experienced deep-RL, first, the rate of each user is determined. Then, these rates are mapped to the resource block and power allocation vectors of the studied wireless system. Finally, the end-to-end reliability and latency of each user are used as feedback to the deep-RL framework. It is then shown that at the fixed-point of the deep-RL algorithm, the reliability and latency of the users are near-optimal. Moreover, for the proposed GAN approach, a theoretical limit for the generator output is analytically derived. Simulation results show how the proposed approach can achieve near-optimal performance within the rate-reliability-latency region, depending on the network and service requirements. The results also show that the proposed experienced deep-RL framework is able to remove the transient training time that makes conventional deep-RL methods unsuitable for URLLC-6G. Moreover, during extreme conditions, it is shown that the proposed, experienced deep-RL agent can recover instantly while a conventional deep-RL agent takes several epochs to adapt to new extreme conditions. [ABSTRACT FROM AUTHOR]

In the current unmanned aircraft systems (UASs) for sensing services, unmanned aerial vehicles (UAVs) transmit their sensory data to terrestrial mobile devices over the unlicensed spectrum. However, the interference from surrounding terminals is uncontrollable due to the opportunistic channel access. In this paper, we consider a cellular Internet of UAVs to guarantee the Quality-of-Service (QoS), where the sensory data can be transmitted to the mobile devices either by UAV-to-Device (U2D) communications over cellular networks, or directly through the base station (BS). Since UAVs’ sensing and transmission may influence their trajectories, we study the trajectory design problem for UAVs in consideration of their sensing and transmission. This is a Markov decision problem (MDP) with a large state-action space, and thus, we utilize multi-agent deep reinforcement learning (DRL) to approximate the state-action space, and then propose a multi-UAV trajectory design algorithm to solve this problem. Simulation results show that our proposed algorithm can achieve a higher total utility than policy gradient algorithm and single-agent algorithm. [ABSTRACT FROM AUTHOR]

Operating LTE networks in unlicensed bands together with legacy WiFi systems is deemed as a promising technique to support explosively growing mobile traffic. In conventional LTE/WiFi spectrum sharing schemes, LTE systems need to know WiFi traffic demands for optimizing system parameters to protect WiFi systems, for which the two systems are required to cooperate with each other via signallings exchanges. However, it is difficult to establish a dedicated channel among the two independent systems for exchanging signallings. Hence, in this paper, we propose an intelligent duty-cycle medium access control protocol to realize the effective and fair spectrum sharing between LTE and WiFi systems without requiring signalling exchanges. Specifically, we first design a duty-cycle spectrum sharing framework, which allows an LTE system to share the spectrum with a WiFi system by using time sharing. After that, we develop deep reinforcement learning (DRL)-based algorithms to learn WiFi traffic demands by analyzing WiFi channel activity, e.g., the idleness/business of WiFi channels, which can be observed by the LTE system via monitoring WiFi channels. Based on the learnt knowledge, the LTE system can adaptively optimize LTE transmission time to maximize its own throughput and meanwhile to provide sufficient protection to the WiFi system. Simulation results show that, in terms of LTE throughput and WiFi protection, the performance of the proposed intelligent scheme can approach that of the genie-aided exhaustive search algorithm, which needs the perfect knowledge of WiFi traffic demands through massive signalling exchanges and is of high computational complexity. [ABSTRACT FROM AUTHOR]

The fifth generation of wireless communications (5G) promises massive increases in traffic volume and data rates, as well as improved reliability in voice calls. Jointly optimizing beamforming, power control, and interference coordination in a 5G wireless network to enhance the communication performance to end users poses a significant challenge. In this paper, we formulate the joint design of beamforming, power control, and interference coordination as a non-convex optimization problem to maximize the signal to interference plus noise ratio (SINR) and solve this problem using deep reinforcement learning. By using the greedy nature of deep Q-learning to estimate future rewards of actions and using the reported coordinates of the users served by the network, we propose an algorithm for voice bearers and data bearers in sub-6 GHz and millimeter wave (mmWave) frequency bands, respectively. The algorithm improves the performance measured by SINR and sum-rate capacity. In realistic cellular environments, the simulation results show that our algorithm outperforms the link adaptation industry standards for sub-6 GHz voice bearers. For data bearers in the mmWave frequency band, our algorithm approaches the maximum sum rate capacity, but with less than 4% of the required run time. [ABSTRACT FROM AUTHOR]

Proposed by Maddah-Ali and Niesen, a coded caching scheme has been verified to alleviate the load of networks efficiently. Recently, a new technique called placement delivery array (PDA) was proposed to characterize the coded caching scheme. In this paper, we consider a caching system in the scope of ultra dense networks (UDNs). Each base station (BS) has a finite cache and stores some contents. We propose an efficient coded content caching scheme called double coded caching to make the transmission robust to in-and-out wireless network quality. Then the dynamic caching and multicast scheduling are considered to jointly minimize the average delay and power of the content-centric wireless networks. This stochastic optimization problem can be formulated as a Markov decision process (MDP) with unknown transition probabilities and large state space. We propose a deep reinforcement learning approach to deal with the decision problem. Our algorithm uses a variational auto-encoder (VAE) neural network to approximate the state sufficiently, and uses a weighted double Q-learning scheme to reduce variance and overestimation of the Q function. Numerical results demonstrate that the proposed double coded caching scheme increases the probability of the successful transmission, and the caching and scheduling policy can effectively reduce the delay and the power consumption. [ABSTRACT FROM AUTHOR]

This paper deals with the use of emerging deep learning techniques in future wireless communication networks. It will be shown that the data-driven approaches should not replace, but rather complement, traditional design techniques based on mathematical models. Extensive motivation is given for why deep learning based on artificial neural networks will be an indispensable tool for the design and operation of future wireless communication networks, and our vision of how artificial neural networks should be integrated into the architecture of future wireless communication networks is presented. A thorough description of deep learning methodologies is provided, starting with the general machine learning paradigm, followed by a more in-depth discussion about deep learning and artificial neural networks, covering the most widely used artificial neural network architectures and their training methods. Deep learning will also be connected to other major learning frameworks, such as reinforcement learning and transfer learning. A thorough survey of the literature on deep learning for wireless communication networks is provided, followed by a detailed description of several novel case studies wherein the use of deep learning proves extremely useful for network design. For each case study, it will be shown how the use of (even approximate) mathematical models can significantly reduce the amount of live data that needs to be acquired/measured to implement the data-driven approaches. Finally, concluding remarks describe those that, in our opinion, are the major directions for future research in this field. [ABSTRACT FROM AUTHOR]