Search

For vehicle-to-network communications, handover (HO) management enables vehicles to maintain the connection with the network while transiting through coverage areas of different base stations (BSs). However, the high mobility of vehicles means shorter connection periods with each BS that leads to frequent HOs, hence raises the necessity for optimal HO decision making for high quality infotainment services. Machine learning is capable of capturing underlying pattern via data driven methods to find optimal solutions to complex problems, and much learning-based HO optimization research has been conducted focusing on specific network setups. However, attention still needs to be paid to the actual deployment aspect and standardized datasets or simulation environments for evaluation. This paper proposes a deep reinforcement learning-based HO algorithm using the input parameters that are configurable in the existing measurement report of cellular networks. The performance of the proposed algorithm is evaluated using the well-known network simulator ns-3 with its official LTE module. A realistic network setup in the city center of Glasgow (U.K.) is configured with vehicle trajectories generated by the routes mobility model using Google Maps Directions API. Evaluation results reveal that the proposed algorithm significantly outperforms the A3 RSRP baseline with an average of 25.72% packet loss reduction per HO, suggesting significant improvement in quality of service of phone call and video streaming, etc. The proposed algorithm also has a small implementation cost compared to some state-of-the-art and should be deployed by a software update to a local BS controller. [ABSTRACT FROM AUTHOR]

To meet the execution requirements of delay-sensitive services in vehicular edge computing (VEC) networks, vehicular services need to be offloaded to edge computing nodes. For complex, large-scale services, the services need to be migrated if the services are not completed before the vehicles leave the coverage of edge computing nodes. Trust and resource matching between areas thus become major problems. This paper studies the decision model of vehicular service offloading and migration. First, software-defined network (SDN) technology is introduced into the traditional network architecture, and a two-layer distributed SDN-controlled VEC network architecture is designed, which is divided into a domain control layer and an area control layer. In this framework, we use the consortium blockchain as a carrier to share network topology information between SDN controllers to prevent information leakage. We then established a service offloading and migration optimization problem model to minimize service execution delay, reduce energy consumption and maximize the throughput of the blockchain system. We describe the problem model as a Markov Decision Process (MDP), introduce a deep reinforcement learning (DRL) algorithm named asynchronous advantage actor-critic (A3C) and design a dynamic service offloading and migration algorithm (DSOMA) based on A3C to solve the problem. Simulation results show that DSOMA can increase the throughput of the blockchain system, and DSOMA is superior to the deep Q-learning (DQN) algorithm and greedy offloading algorithm in reducing service execution delay and system energy consumption. [ABSTRACT FROM AUTHOR]

The Internet of vehicles (IoV) is a large information interaction network that collects information on vehicles, roads and pedestrians. One of the important uses of vehicle networks is to meet the entertainment needs of driving users through communication between vehicles and roadside units (RSUs). Due to the limited storage space of RSUs, determining the content cached in each RSU is a key challenge. With the development of 5G and video editing technology, short video systems have become increasingly popular. Current widely used cache update methods, such as partial file precaching and content popularity- and user interest-based determination, are inefficient for such systems. To solve this problem, this paper proposes a QoE-driven edge caching method for the IoV based on deep reinforcement learning. First, a class-based user interest model is established. Compared with the traditional file popularity- and user interest distribution-based cache update methods, the proposed method is more suitable for systems with a large number of small files. Second, a quality of experience (QoE)-driven RSU cache model is established based on the proposed class-based user interest model. Third, a deep reinforcement learning method is designed to address the QoE-driven RSU cache update issue effectively. The experimental results verify the effectiveness of the proposed algorithm. [ABSTRACT FROM AUTHOR]

Both Device-to-device (D2D) and full-duplex (FD) have been widely recognized as spectrum efficient techniques in the fifth-generation (5G) networks. By combining them, the FD-D2D aided underlaying networks (FN) has exhibited considerable technical advantages in terms of both spectral efficiency (SE) and energy efficiency (EE). Considering the fact that the performance of FN may be severely affected by users’ workload, the workload-driven FN (WFN) must be investigated. In this paper, a deep learning based transmit power allocation (TPA) method is proposed for automatically determining the optimal transmit powers of co-spectrum cellular users (CUs) and D2D users (DUs) relying on a deep neural network. Unlike the conventional transmit-power-control schemes, in which complex optimization problems must be addressed in an iterative manner (it usually requires a relative longer computational time), the proposed scheme enables each DU to determine its transmit power with a relatively shorter time. Furthermore, an improved iterative subspace-pursuit algorithm, as the performance benchmark, is formulated for WFN. In addition, to reflect the influence imposed by the workload, the penalty-based statistical sum-date-rate (PSS) can be employed as the performance metric of WFN. Numerical results show that the proposed scheme is capable of achieving a PSS comparable with that of the traditional iterative-based algorithms even under heavy-workload scenarios, but the computational complexity of the former can be significantly reduced. [ABSTRACT FROM AUTHOR]

In this paper, we study the handover (HO), and power allocation problem in a two-tier heterogeneous network (HetNet), which consists of a macro base station, and some millimeter-wave (mmWave) small base stations. We establish an HO management, and power allocation scheme to maximize the overall throughput while reducing the HO frequency. In particular, considering the interrelationship among decisions made by different user equipments (UEs), we first model the HO, and power allocation problem as a fully cooperative multi-agent task, in which all agents, i.e., UEs, have the same target. Then, to solve the multi-agent task, and get decentralized policies for each UE, we develop a multi-agent reinforcement learning (MARL) algorithm based on the proximal policy optimization (PPO) method, by introducing the centralized training with decentralized execution framework. That is, we use global information to train policies for each UE, and after the training is finished, each UE obtains a decentralized policy, which can be implemented only based on each UE's local observation. Specially, we introduce the counterfactual baseline to address the credit assignment problem in centralized learning. Due to the centralized training, the decentralized polices learned by multi-agent PPO (MAPPO) can work more cooperatively. Finally, the simulation results demonstrate that our method can achieve better performance comparing with other existing works. [ABSTRACT FROM AUTHOR]

The vehicular edge computing is considered an enabling technology for intelligent and connected vehicles since the optimization of communication and computing on edge has a significant impact on driving safety and efficiency. In this paper, with the road traffic assignment to “proactively” reshape the spatiotemporal distribution of resource demands, we investigate the joint service migration and mobility optimization problem for vehicular edge computing. The goal is to meet the service delay requirements of vehicular edge computing with minimum migration cost and travel time. As service migration and mobility optimization are coupled, the joint scheduling problem suffers from the curse of dimensionality, which cannot be solved in real time by centralized algorithms. To this end, a multi-agent deep reinforcement learning (MADRL) algorithm is proposed to maximize the composite utility of communication, computing, and route planning in a distributed way. In the MADRL algorithm, a two-branch convolution based deep Q-network is constructed to coordinate migration action and routing action. Extensive experimental results show that the proposed algorithm is scalable and substantially reduces service delay, migration cost and travel time as compared with the existing baselines. [ABSTRACT FROM AUTHOR]

As urban traffic condition is diverse and complicated, applying reinforcement learning to reduce traffic congestion becomes one of the hot and promising topics. Especially, how to coordinate the traffic light controllers of multiple intersections is a key challenge for multi-agent reinforcement learning (MARL). Most existing MARL studies are based on traditional $Q$ -learning, but unstable environment leads to poor learning in the complicated and dynamic traffic scenarios. In this paper, we propose a novel multi-agent recurrent deep deterministic policy gradient (MARDDPG) algorithm based on deep deterministic policy gradient (DDPG) algorithm for traffic light control (TLC) in vehiclar networks. Specifically, the centralized learning in each critic network enables each agent to estimate the policies of other agents in the decision-making process and each agent can coordinate with each other, alleviating the problem of poor learning performance caused by environmental instability. The decentralized execution enables each agent to make decisions independently. We share parameters in actor networks to speed up the training process and reduce the memory footprint. The addition of LSTM is beneficial to alleviate the instability of the environment caused by partial observable state. We utilize surveillance cameras and vehicular networks to collect status information for each intersection. Unlike previous work, we have not only considered the vehicle but also considered the pedestrians waiting to pass through the intersection. Moreover, we also set different priorities for buses and ordinary vehicles. The experimental results in a vehicular network show that our method can run stably in various scenarios and coordinate multiple intersections, which significantly reduces vehicle congestion and pedestrian congestion. [ABSTRACT FROM AUTHOR]

Wireless virtual reality (VR) is predicted to become a killer application in 5G and beyond, which provides an immersive experience and revolutionizes the way people communicate. It is well-known that rendering is the key performance bottleneck in wireless VR systems, especially for VR games. However, real-time rendering and data correlation are ignored by most researchers. In this paper, we propose an adaptive VR framework that enables high-quality wireless VR in future mmWave-enabled wireless networks with mobile edge computing (MEC), where real-time VR rendering tasks can be offloaded to MEC servers adaptively and the caching capability of MEC servers enables further performance improvement. First, we formulate the addressed problem to maximize the quality of experience (QoE) of the users, where association, adaptive offloading mode selection, and caching policy are jointly optimized. Considering the high complexity of the addressed problem, we then propose a distributed learning approach consisting of an offline training phase and an online running phase, which maintains scalability and adaptation capability. The offline phase is based on deep reinforcement learning (DRL) while the latter utilizes game theory. At last, simulation results show the superiority of the proposed algorithm over the other baseline algorithms in terms of QoE utility values, latency, and convergence time. [ABSTRACT FROM AUTHOR]

Unmanned Aerial Vehicles (UAVs) with Microwave Power Transfer (MPT) capability provide a practical means to deploy a large number of wireless powered sensing devices into areas with no access to persistent power supplies. The UAV can charge the sensing devices remotely and harvest their data. A key challenge is online MPT and data collection in the presence of on-board control of a UAV (e.g., patrolling velocity) for preventing battery drainage and data queue overflow of the devices, while up-to-date knowledge on battery level and data queue of the devices is not available at the UAV. In this paper, an on-board deep Q-network is developed to minimize the overall data packet loss of the sensing devices, by optimally deciding the device to be charged and interrogated for data collection, and the instantaneous patrolling velocity of the UAV. Specifically, we formulate a Markov Decision Process (MDP) with the states of battery level and data queue length of devices, channel conditions, and waypoints given the trajectory of the UAV; and solve it optimally with Q-learning. Furthermore, we propose the on-board deep Q-network that enlarges the state space of the MDP, and a deep reinforcement learning based scheduling algorithm that asymptotically derives the optimal solution online, even when the UAV has only outdated knowledge on the MDP states. Numerical results demonstrate that our deep reinforcement learning algorithm reduces the packet loss by at least 69.2%, as compared to existing non-learning greedy algorithms. [ABSTRACT FROM AUTHOR]

Energy efficiency (EE) and spectrum efficiency (SE) have received significant attentions on optimizing the network performance in cognitive radio networks. In this paper, an EE+SE tradeoff based target is considered for the primary users (PUs) and the secondary users (SUs). First of all, considering the orthogonal frequency division multiple access-based resource allocation (RA) for the underlying SUs, we formulate an objective function through minimizing a weighted sum of the secondary interference power, where the network performance of both PUs and SUs are guaranteed by the constraints on quality of service, power consumption and data rate. However, it is a NP-hard problem. In order to solve it, we propose a damped three dimensional (D3D) message-passing algorithm (MPA) based on deep learning. Specifically, a feed-forward neural network is devised and an analogous back propagation algorithm is developed to learn the optimal parameters of the D3D-MPA. To improve the computational efficiency of the allocation and the learning, a suboptimal RA scheme is deduced based on a damped two dimensional MPA. Finally, simulation results are provided to confirm the effectiveness of our proposed scheme. [ABSTRACT FROM AUTHOR]