Search

The Internet of vehicles (IoV) is a large information interaction network that collects information on vehicles, roads and pedestrians. One of the important uses of vehicle networks is to meet the entertainment needs of driving users through communication between vehicles and roadside units (RSUs). Due to the limited storage space of RSUs, determining the content cached in each RSU is a key challenge. With the development of 5G and video editing technology, short video systems have become increasingly popular. Current widely used cache update methods, such as partial file precaching and content popularity- and user interest-based determination, are inefficient for such systems. To solve this problem, this paper proposes a QoE-driven edge caching method for the IoV based on deep reinforcement learning. First, a class-based user interest model is established. Compared with the traditional file popularity- and user interest distribution-based cache update methods, the proposed method is more suitable for systems with a large number of small files. Second, a quality of experience (QoE)-driven RSU cache model is established based on the proposed class-based user interest model. Third, a deep reinforcement learning method is designed to address the QoE-driven RSU cache update issue effectively. The experimental results verify the effectiveness of the proposed algorithm. [ABSTRACT FROM AUTHOR]

Deep reinforcement learning (DRL) is a promising technology to address the resource allocation problem for efficient data transmission in complex network environments. However, most DRL-based resource allocation algorithms suffer from limited feature extraction capabilities and lack scalability and generalization, especially in heterogeneous Industrial Internet of Things (IIoT) environments. In this paper, we develop a lifelong learning architecture that can integrate artificial intelligence (AI) algorithms into the heterogeneous IIoT network for efficient data transmission. Based on this, we propose an intelligent resource allocation algorithm based on dual-attention DRL (DADR) for forwarding node selection and channel access slot allocation in a specific network environment. The proposed DADR algorithm combines the advantages of multi-dimension convolutional attention and multi-head self-attention mechanisms. It can provide local- and global-feature fusion capabilities for distributed nodes while maximizing the performance of data transmission. Furthermore, we present a lifelong federated meta reinforcement learning (LFMRL) that can effectively utilize prior knowledge and enable the DRL agent quickly adapt to a new environment. Specifically, LFMRL adopts a federated meta learning-based knowledge fusion algorithm to fuse the knowledge of learned DADR algorithms and iteratively update the shared model, thereby improving the scalability and generalization of the shared model in heterogeneous IIoT environments. In addition, a simple and efficient knowledge transfer mechanism is enabled to accelerate the DRL model convergence by transferring the knowledge of the shared model to the new environment. Simulation results demonstrate the effectiveness of the proposed algorithms in terms of energy efficiency, data transmission reliability, and network stability. Compared to DADR and FedAvg algorithms, LFMRL algorithm can further reduce the energy consumption, training time, and average forwarding node switching times, while improving packet delivery rate to 99.2%. • A novel lifelong learning architecture for resource allocation in heterogeneous IIoT. • Dual-attention deep reinforcement learning provides local- and global- feature fusion. • Federated meta learning based knowledge fusion learns the commonness of models. • Reinforcement learning based knowledge transfer learns the properties of model. • Knowledge fusion and knowledge transfer provide lifelong learning ability for system. [ABSTRACT FROM AUTHOR]

As urban traffic condition is diverse and complicated, applying reinforcement learning to reduce traffic congestion becomes one of the hot and promising topics. Especially, how to coordinate the traffic light controllers of multiple intersections is a key challenge for multi-agent reinforcement learning (MARL). Most existing MARL studies are based on traditional $Q$ -learning, but unstable environment leads to poor learning in the complicated and dynamic traffic scenarios. In this paper, we propose a novel multi-agent recurrent deep deterministic policy gradient (MARDDPG) algorithm based on deep deterministic policy gradient (DDPG) algorithm for traffic light control (TLC) in vehiclar networks. Specifically, the centralized learning in each critic network enables each agent to estimate the policies of other agents in the decision-making process and each agent can coordinate with each other, alleviating the problem of poor learning performance caused by environmental instability. The decentralized execution enables each agent to make decisions independently. We share parameters in actor networks to speed up the training process and reduce the memory footprint. The addition of LSTM is beneficial to alleviate the instability of the environment caused by partial observable state. We utilize surveillance cameras and vehicular networks to collect status information for each intersection. Unlike previous work, we have not only considered the vehicle but also considered the pedestrians waiting to pass through the intersection. Moreover, we also set different priorities for buses and ordinary vehicles. The experimental results in a vehicular network show that our method can run stably in various scenarios and coordinate multiple intersections, which significantly reduces vehicle congestion and pedestrian congestion. [ABSTRACT FROM AUTHOR]

The chemical mechanical polishing (CMP) process usually suffers from drift and shift in the Run-to-Run material removal process due to the wear and replacement of the polishing pad, lacking of in-suit measurements of the product quality of interest and other environment variations. This paper proposed a deep reinforcement learning (DRL)-based run-to-run (R2R) controller for the CMP process. Firstly, deep reinforcement learning is effectively utilized as a training algorithm of the R2R controller, which is a model-free controller to take a decision with infinite horizon information and thus improves the control performance; Secondly, a novel policy network is embeded to the DRL model, which divides the network into linear and nonlinear part explicitly to improve the prediction performance of the R2R controller on process changes. Finally, a special reward function is proposed to improve the training of the R2R controller, which trades off between target tracing and fluctuations of production parameters. The effectiveness of the proposed controller is validated on a CMP process. The testing results illustrate that the DRL-based R2R controller can precisely trace the desired target of material removal rate (MRR) and is very effective to control various process variations online. [ABSTRACT FROM AUTHOR]

In the current unmanned aircraft systems (UASs) for sensing services, unmanned aerial vehicles (UAVs) transmit their sensory data to terrestrial mobile devices over the unlicensed spectrum. However, the interference from surrounding terminals is uncontrollable due to the opportunistic channel access. In this paper, we consider a cellular Internet of UAVs to guarantee the Quality-of-Service (QoS), where the sensory data can be transmitted to the mobile devices either by UAV-to-Device (U2D) communications over cellular networks, or directly through the base station (BS). Since UAVs’ sensing and transmission may influence their trajectories, we study the trajectory design problem for UAVs in consideration of their sensing and transmission. This is a Markov decision problem (MDP) with a large state-action space, and thus, we utilize multi-agent deep reinforcement learning (DRL) to approximate the state-action space, and then propose a multi-UAV trajectory design algorithm to solve this problem. Simulation results show that our proposed algorithm can achieve a higher total utility than policy gradient algorithm and single-agent algorithm. [ABSTRACT FROM AUTHOR]

Proposed by Maddah-Ali and Niesen, a coded caching scheme has been verified to alleviate the load of networks efficiently. Recently, a new technique called placement delivery array (PDA) was proposed to characterize the coded caching scheme. In this paper, we consider a caching system in the scope of ultra dense networks (UDNs). Each base station (BS) has a finite cache and stores some contents. We propose an efficient coded content caching scheme called double coded caching to make the transmission robust to in-and-out wireless network quality. Then the dynamic caching and multicast scheduling are considered to jointly minimize the average delay and power of the content-centric wireless networks. This stochastic optimization problem can be formulated as a Markov decision process (MDP) with unknown transition probabilities and large state space. We propose a deep reinforcement learning approach to deal with the decision problem. Our algorithm uses a variational auto-encoder (VAE) neural network to approximate the state sufficiently, and uses a weighted double Q-learning scheme to reduce variance and overestimation of the Q function. Numerical results demonstrate that the proposed double coded caching scheme increases the probability of the successful transmission, and the caching and scheduling policy can effectively reduce the delay and the power consumption. [ABSTRACT FROM AUTHOR]

Unmanned Aerial Vehicles (UAVs) with Microwave Power Transfer (MPT) capability provide a practical means to deploy a large number of wireless powered sensing devices into areas with no access to persistent power supplies. The UAV can charge the sensing devices remotely and harvest their data. A key challenge is online MPT and data collection in the presence of on-board control of a UAV (e.g., patrolling velocity) for preventing battery drainage and data queue overflow of the devices, while up-to-date knowledge on battery level and data queue of the devices is not available at the UAV. In this paper, an on-board deep Q-network is developed to minimize the overall data packet loss of the sensing devices, by optimally deciding the device to be charged and interrogated for data collection, and the instantaneous patrolling velocity of the UAV. Specifically, we formulate a Markov Decision Process (MDP) with the states of battery level and data queue length of devices, channel conditions, and waypoints given the trajectory of the UAV; and solve it optimally with Q-learning. Furthermore, we propose the on-board deep Q-network that enlarges the state space of the MDP, and a deep reinforcement learning based scheduling algorithm that asymptotically derives the optimal solution online, even when the UAV has only outdated knowledge on the MDP states. Numerical results demonstrate that our deep reinforcement learning algorithm reduces the packet loss by at least 69.2%, as compared to existing non-learning greedy algorithms. [ABSTRACT FROM AUTHOR]

This paper addresses the challenge of minimizing training time for the control of Heating, Ventilation, and Air-conditioning (HVAC) systems with online Reinforcement Learning (RL). This is done by developing a novel approach to Multi-Agent Reinforcement Learning (MARL) to HVAC systems. In this paper, the environment formed by the HVAC system is formulated as a Markov Game (MG) in a general sum setting. The MARL algorithm is designed in a decentralized structure, where only relevant states are shared between agents, and actions are shared in a sequence, which are sensible from a system's point of view. The simulation environment is a domestic house located in Denmark and designed to resemble an average house. The heat source in the house is an air-to-water heat pump, and the HVAC system is an Underfloor Heating system (UFH). The house is subjected to weather changes from a data set collected in Copenhagen in 2006, spanning the entire year except for June, July, and August, where heat is not required. It is shown that: (1) When comparing Single Agent Reinforcement Learning (SARL) and MARL, training time can be reduced by 70% for a four temperature-zone UFH system, (2) the agent can learn and generalize over seasons, (3) the cost of heating can be reduced by 19% or the equivalent to 750 kWh of electric energy per year for an average Danish domestic house compared to a traditional control method, and (4) oscillations in the room temperature can be reduced by 40% when comparing the RL control methods with a traditional control method. [ABSTRACT FROM AUTHOR]

A pursuit–evasion game is a classical maneuver confrontation problem in the multi-agent systems (MASs) domain. An online decision technique based on deep reinforcement learning (DRL) was developed in this paper to address the problem of environment sensing and decision-making in pursuit–evasion games. A control-oriented framework developed from the DRL-based multi-agent deep deterministic policy gradient (MADDPG) algorithm was built to implement multi-agent cooperative decision-making to overcome the limitation of the tedious state variables required for the traditionally complicated modeling process. To address the effects of errors between a model and a real scenario, this paper introduces adversarial disturbances. It also proposes a novel adversarial attack trick and adversarial learning MADDPG (A2-MADDPG) algorithm. By introducing an adversarial attack trick for the agents themselves, uncertainties of the real world are modeled, thereby optimizing robust training. During the training process, adversarial learning was incorporated into our algorithm to preprocess the actions of multiple agents, which enabled them to properly respond to uncertain dynamic changes in MASs. Experimental results verified that the proposed approach provides superior performance and effectiveness for pursuers and evaders, and both can learn the corresponding confrontational strategy during training. [ABSTRACT FROM AUTHOR]

In an autonomous vehicle, the lane following algorithm is an important component, which is a basic function of autonomous driving. However, the existing lane following system has a few shortcomings: first, the control method it adopts requires an accurate system model, and different vehicles have different parameters, which needs a lot of parameter calibration work. The second is that it may fail on road sections where the lateral acceleration requirements of vehicles are large, such as large curves. Third, its decision-making system is defined based on rules, which has disadvantages: it is difficult to formulate; human subjective factors cannot guarantee objectivity; coverage is difficult to guarantee. In recent years, the deep deterministic policy gradient (DDPG) algorithm has been widely used in the field of autonomous driving due to its strong nonlinear fitting ability and generalization performance. However, the DDPG algorithm has overestimated state action values and large cumulative errors, low training efficiency and other issues. Therefore, this paper improves the DDPG algorithm based on the double critic networks and priority experience replay mechanism. Then this paper proposes a lane following method based on this algorithm. Experiment shows that the algorithm can achieve excellent following results under various road conditions. [ABSTRACT FROM AUTHOR]

This paper proposes an optimal route and charging station selection (RCS) algorithm based on model-free deep reinforcement learning (DRL) to overcome the uncertainty issues of the traffic conditions and dynamic arrival charging requests. The proposed DRL based RCS algorithm aims to minimize the total travel time of electric vehicles (EV) charging requests from origin to destination using the selection of the optimal route and charging station considering dynamically changing traffic conditions and unknown future requests. In this paper, we formulate this RCS problem as a Markov decision process model with unknown transition probability. A Deep Q network has been adopted with function approximation to find the optimal electric vehicle charging station (EVCS) selection policy. To obtain the feature states for each EVCS, we define the traffic preprocess module, charging preprocess module and feature extract module. The proposed DRL based RCS algorithm is compared with conventional strategies such as minimum distance, minimum travel time, and minimum waiting time. The performance is evaluated in terms of travel time, waiting time, charging time, driving time, and distance under the various distributions and number of EV charging requests. [ABSTRACT FROM AUTHOR]

Behind-the-meter distributed energy resources (DERs), including building solar photovoltaic (PV) technology and electric battery storage, are increasingly being considered as solutions to support carbon reduction goals and increase grid reliability and resiliency. However, dynamic control of these resources in concert with traditional building loads, to effect efficiency and demand flexibility, is not yet commonplace in commercial control products. Traditional rule-based control algorithms do not offer integrated closed-loop control to optimize across systems, and most often, PV and battery systems are operated for energy arbitrage and demand charge management, and not for the provision of grid services. More advanced control approaches, such as MPC control have not been widely adopted in industry because they require significant expertise to develop and deploy. Recent advances in deep reinforcement learning (DRL) offer a promising option to optimize the operation of DER systems and building loads with reduced setup effort. However, there are limited studies that evaluate the efficacy of these methods to control multiple building subsystems simultaneously. Additionally, most of the research has been conducted in simulated environments as opposed to real buildings. This paper proposes a DRL approach that uses a deep deterministic policy gradient algorithm for integrated control of HVAC and electric battery storage systems in the presence of on-site PV generation. The DRL algorithm, trained on synthetic data, was deployed in a physical test building and evaluated against a baseline that uses the current best-in-class rule-based control strategies. Performance in delivering energy efficiency, load shift, and load shed was tested using price-based signals. The results showed that the DRL-based controller can produce cost savings of up to 39.6% as compared to the baseline controller, while maintaining similar thermal comfort in the building. The project team has also integrated the simulation components developed during this work as an OpenAIGym environment and made it publicly available so that prospective DRL researchers can leverage this environment to evaluate alternate DRL algorithms. • Traditional controls do not integrate distributed energy resources (DER) systems. • Deterministic policy gradient algorithm is proposed to optimize the operation of DER. • The algorithm is deployed and evaluated in a physical test building. • Tests include energy efficiency, load shift, and load shed. • An OpenAI Gym environment is made it publicly available for other researchers. [ABSTRACT FROM AUTHOR]

Wireless networking using GHz or THz spectra has encouraged mobile service providers to deploy small cells to improve link quality and cell capacity using mmWave backhaul links. As green networking for less CO2 emission is mandatory to confront global climate change, we need energy efficient network management for such denser small-cell heterogeneous networks (HetNets) that already suffer from observable power consumption. We establish a dual-objective optimization model that minimizes energy consumption by switching off unused small cells while maximizing user throughput, which is a mixed integer linear problem (MILP). Recently, the deep reinforcement learning (DRL) algorithm has been applied to many NP-hard problems of the wireless networking field, such as radio resource allocation, association and power saving, which can induce a near-optimal solution with fast inference time as an online solution. In this paper, we investigate the feasibility of the DRL algorithm for a dual-objective problem, energy efficient routing and throughput maximization, which has not been explored before. We propose a proximal policy (PPO)-based multi-objective algorithm using the actor-critic model that is realized as an optimistic linear support framework in which the PPO algorithm searches for feasible solutions iteratively. Experimental results show that our algorithm can achieve throughput and energy savings comparable to the CPLEX. [ABSTRACT FROM AUTHOR]

Earth observation satellite task scheduling research plays a key role in space-based remote sensing services. An effective task scheduling strategy can maximize the utilization of satellite resources and obtain larger objective observation profits. In this paper, inspired by the success of deep reinforcement learning in optimization domains, the deep deterministic policy gradient algorithm is adopted to solve a time-continuous satellite task scheduling problem. Moreover, an improved graph-based minimum clique partition algorithm is proposed for preprocessing in the task clustering phase by considering the maximum task priority and the minimum observation slewing angle under constraint conditions. Experimental simulation results demonstrate that the deep reinforcement learning-based task scheduling method is feasible and performs much better than traditional metaheuristic optimization algorithms, especially in large-scale problems. [ABSTRACT FROM AUTHOR]

Because of the high penetration of renewable energies and the installation of new control devices, modern distribution networks are faced with voltage regulation challenges. Recently, the rapid development of artificial intelligence technology has introduced new solutions for optimal control problems with high dimensions and dynamics. In this paper, a deep reinforcement learning method is proposed to solve the two-timescale optimal voltage control problem. All control variables are assigned to different agents, and discrete variables are solved by a deep Q network (DQN) agent while the continuous variables are solved by a deep deterministic policy gradient (DDPG) agent. All agents are trained simultaneously with specially designed reward aiming at minimizing long-term average voltage deviation. Case study is executed on a modified IEEE-123 bus system, and the results demonstrate that the proposed algorithm has similar or even better performance than the model-based optimal control scheme and has high computational efficiency and competitive potential for online application. [ABSTRACT FROM AUTHOR]

How to operate an unmanned aerial vehicle (UAV) safely and efficiently in an interactive environment is challenging. A large amount of research has been devoted to improve the intelligence of a UAV while performing a mission, where finding an optimal maneuver decision-making policy of the UAV has become one of the key issues when we attempt to enable the UAV autonomy. In this paper, we propose a maneuver decision-making algorithm based on deep reinforcement learning, which generates efficient maneuvers for a UAV agent to execute the airdrop mission autonomously in an interactive environment. Particularly, the training set of the learning algorithm by the Prioritized Experience Replay is constructed, that can accelerate the convergence speed of decision network training in the algorithm. It is shown that a desirable and effective maneuver decision-making policy can be found by extensive experimental results. [ABSTRACT FROM AUTHOR]

• Online network reconfiguration is introduced to manage distribution networks. • A Deep Q-learning algorithm is used to find the optimal network topology. • The proposed algorithm is more scalable than other algorithms. • The computation time of the algorithm is low enough for practical applications. • The algorithm can increase DRES utilization without heavy capital investment. With increasing number of distributed renewable energy sources integrated in power distribution networks, network security issues such as line overloading or bus voltage violations are becoming increasingly common. Traditional capital-intensive system reinforcements could lead to overinvestment. Moreover, active network management solutions, which have emerged as important alternatives, may become a financial burden for distribution system operators or reduce profits for owners of distributed renewable energy sources, or both. To address these limitations, this paper proposes an online network reconfiguration scheme based on a deep reinforcement learning approach. In this scheme, the distribution network operator modifies the network topology to change the power flow when the reliability of network is threatened. Because the variability of distributed renewable energy is large in self-sufficient distribution networks, the reconfiguration process needs to be performed online within short time intervals, which involves the use of conventional algorithms. To solve this problem efficiently, a deep q-learning model is utilized to determine the optimal network topology. Performances of proposed and other algorithms were compared in modified CIGRE 14-bus and IEEE 123-bus test network, as well as varying penalties for frequent switching operation in consideration of physical characteristic of the network. Simulation results demonstrated that the proposed algorithm showed almost identical performances with brute-force search algorithm in both test networks, satisfying network constraints over almost all timespans. Further, the proposed method required very small computation times - under a second per each state and its scalability was verified by comparing the computation time between two test networks. [ABSTRACT FROM AUTHOR]

• Apply a state-of-the-art deep reinforcement learning algorithm to the PM problem. • Formulate the PM problem as an MDP with guidance of domain knowledge. • Use a data-driven modeling method to build a fast simulator for learning. • The DRL agent learns to perform group maintenance and opportunistic maintenance. • Proposed method outperforms age-dependent policy and opportunistic policy. In the manufacturing industry, the preventive maintenance (PM) is a common practice to reduce random machine failures by replacing/repairing the aged machines or parts. The decision on when and where the preventive maintenance needs to be carried out is nontrivial due to the complex and stochastic nature of a serial production line with intermediate buffers. In order to improve the cost efficiency of the serial production lines, a deep reinforcement learning based approach is proposed to obtain PM policy. A novel modeling method for the serial production line is adopted during the learning process. A reward function is proposed based on the system production loss evaluation. The algorithm based on the Double Deep Q-Network is applied to learn the PM policy. Using the simulation study, the learning algorithm is proved effective in delivering PM policy that leads to an increased throughput and reduced cost. Interestingly, the learned policy is found to frequently conduct "group maintenance" and "opportunistic maintenance", although their concepts and rules are not provided during the learning process. This finding further demonstrates that the problem formulation, the proposed algorithm and the reward function setting in this paper are effective. [ABSTRACT FROM AUTHOR]

In this paper, a model-free deep reinforcement learning (DRL) strategy is presented with an artificial neural network (ANN) as reaction simulation environment, to obtain a fed-batch control strategy for an experimental bioreactor. The proposed method is a fundamental attempt to control reactions by employing state-of-the-art machine learning tools without the aid of well-established mechanistic understanding of the reaction system. This application utilizes the Asynchronous Advantage Actor-Critic (A3C) algorithm, a member of the DRL family, that takes advantage of actor-critic algorithm and asynchronous learning by parallel learning agents to achieve stability and efficiency of the learning process. The resulting controller demonstrates robust performance in the fed-batch bioreactor since it can be adjusted to meet varying constraining factors including nutrient limitations and culture lengths. Results are presented for a bioreactor that produces cyanobacterial-phycocyanin (C-PC) in Plectonema sp. UTEX 1541. Experimental validations show a 52.1% increase in the product yield, and a 20.1% increase in C-PC concentration compared to a control group with the same total nutrient input replenished in a non-optimized manner. [ABSTRACT FROM AUTHOR]

With advances in smart grid technologies, demand response has played a major role in improving the reliability of grids and reduce the cost for customers. Implementing the demand response scheme for industry is more necessary than for other sectors, because its energy consumption is often considered the largest. This paper proposes a multi-agent deep reinforcement learning based demand response scheme for energy management of discrete manufacturing systems. In this regard, the industrial manufacturing system is initially formulated as a partially-observable Markov game; then, a multi-agent deep deterministic policy gradient algorithm is adopted to obtain the optimal schedule for different machines. A typical lithium-ion battery assembly manufacturing system is used to demonstrate the effectiveness of the proposed scheme. Simulation results show that the presented demand response algorithm can minimize electricity costs and maintain production tasks, as compared to a benchmark without demand response. Moreover, the performance of the multi-agent deep reinforcement learning approach against a mathematical model method is investigated. [ABSTRACT FROM AUTHOR]

• Control framework based on deep-Q-network is developed. • Monte-Carlo sampling method is integrated to generate simulation datasets. • The proposed model is applied to the extraction column in downstream process. • Interface based on component object model is suggested to integrate algorithms. This paper proposes a systematic framework to develop deep reinforcement learning (RL)-based algorithms for control system of downstream separation in biopharmaceutical process as follows. First, a simulation model as a digital twin is built and Monte-Carlo sampling generates substantial amounts of samples considering disturbances. Second, the deep RL-based control system is designed and the optimization subject to sample datasets is conducted. The methodology is implemented in a prototype software and relevant codes are shared by Mendeley Data. The proposed model is successfully applied to control the liquid-liquid extraction column for the recovery of fusidic acid as part of downstream processing. The resulting deep RL algorithm provides an operation performance with a better API recovery yield (32 % higher than open loop operation) and lower deviations (23 % lower than open loop operation) against disturbances. Image, graphical abstract [ABSTRACT FROM AUTHOR]

To date, automatic driving technology has become a hotspot in academia. It is necessary to provide a personalization of automatic driving decision for each passenger. The purpose of this paper is to propose a self-learning method for personalized driving decisions. First, collect and analyze driving data from different drivers to set learning goals. Then, Deep Deterministic Policy Gradient algorithm is utilized to design a driving decision system. Furthermore, personalized factors are introduced for some observed parameters to build a personalized driving decision model. Finally, compare the proposed method with classic Deep Reinforcement Learning algorithms. The results show that the performance of the personalized driving decision model is better than the classic algorithms, and it is similar to the manual driving situation. Therefore, the proposed model can effectively learn the human-like personalized driving decisions of different drivers for structured road. Based on this model, the smart car can accomplish personalized driving. [ABSTRACT FROM AUTHOR]

It is crucial for robots to autonomously steer in complex environments safely without colliding with any obstacles. Compared to conventional methods, deep reinforcement learning-based methods are able to learn from past experiences automatically and enhance the generalization capability to cope with unseen circumstances. Therefore, we propose an end-to-end deep reinforcement learning algorithm in this paper to improve the performance of autonomous steering in complex environments. By embedding a branching noisy dueling architecture, the proposed model is capable of deriving steering commands directly from raw depth images with high efficiency. Specifically, our learning-based approach extracts the feature representation from depth inputs through convolutional neural networks and maps it to both linear and angular velocity commands simultaneously through different streams of the network. Moreover, the training framework is also meticulously designed to improve the learning efficiency and effectiveness. It is worth noting that the developed system is readily transferable from virtual training scenarios to real-world deployment without any fine-tuning by utilizing depth images. The proposed method is evaluated and compared with a series of baseline methods in various virtual environments. Experimental results demonstrate the superiority of the proposed model in terms of average reward, learning efficiency, success rate as well as computational time. Moreover, a variety of real-world experiments are also conducted which reveal the high adaptability of our model to both static and dynamic obstacle-cluttered environments. [ABSTRACT FROM AUTHOR]