Search

Efficient point cloud compression is essential for applications like virtual and mixed reality, autonomous driving, and cultural heritage. This paper proposes a deep learning-based inter-frame encoding scheme for dynamic point cloud geometry compression. We propose a lossy geometry compression scheme that predicts the latent representation of the current frame using the previous frame by employing a novel feature space inter-prediction network. The proposed network utilizes sparse convolutions with hierarchical multiscale 3D feature learning to encode the current frame using the previous frame. The proposed method introduces a novel predictor network for motion compensation in the feature domain to map the latent representation of the previous frame to the coordinates of the current frame to predict the current frame’s feature embedding. The framework transmits the residual of the predicted features and the actual features by compressing them using a learned probabilistic factorized entropy model. At the receiver, the decoder hierarchically reconstructs the current frame by progressively rescaling the feature embedding. The proposed framework is compared to the state-of-the-art Video-based Point Cloud Compression (V-PCC) and Geometry-based Point Cloud Compression (G-PCC) schemes standardized by the Moving Picture Experts Group (MPEG). The proposed method achieves more than 88% BD-Rate (Bjøntegaard Delta Rate) reduction against G-PCCv20 Octree, more than 56% BD-Rate savings against G-PCCv20 Trisoup, more than 62% BD-Rate reduction against V-PCC intra-frame encoding mode, and more than 52% BD-Rate savings against V-PCC P-frame-based inter-frame encoding mode using HEVC. These significant performance gains are cross-checked and verified in the MPEG working group. [ABSTRACT FROM AUTHOR]

Estimating depth from RGB images is a long-standing ill-posed problem, which has been explored for decades by the computer vision, graphics, and machine learning communities. Among the existing techniques, stereo matching remains one of the most widely used in the literature due to its strong connection to the human binocular system. Traditionally, stereo-based depth estimation has been addressed through matching hand-crafted features across multiple images. Despite the extensive amount of research, these traditional techniques still suffer in the presence of highly textured areas, large uniform regions, and occlusions. Motivated by their growing success in solving various 2D and 3D vision problems, deep learning for stereo-based depth estimation has attracted a growing interest from the community, with more than 150 papers published in this area between 2014 and 2019. This new generation of methods has demonstrated a significant leap in performance, enabling applications such as autonomous driving and augmented reality. In this paper, we provide a comprehensive survey of this new and continuously growing field of research, summarize the most commonly used pipelines, and discuss their benefits and limitations. In retrospect of what has been achieved so far, we also conjecture what the future may hold for deep learning-based stereo for depth estimation research. [ABSTRACT FROM AUTHOR]

3D neural networks are widely used in real-world applications (e.g., AR/VR headsets, self-driving cars). They are required to be fast and accurate; however, limited hardware resources on edge devices make these requirements rather challenging. Previous work processes 3D data using either voxel-based or point-based neural networks, but both types of 3D models are not hardware-efficient due to the large memory footprint and random memory access. In this paper, we study 3D deep learning from the efficiency perspective. We first systematically analyze the bottlenecks of previous 3D methods. We then combine the best from point-based and voxel-based models together and propose a novel hardware-efficient 3D primitive, Point-Voxel Convolution (PVConv). We further enhance this primitive with the sparse convolution to make it more effective in processing large (outdoor) scenes. Based on our designed 3D primitive, we introduce 3D Neural Architecture Search (3D-NAS) to explore the best 3D network architecture given a resource constraint. We evaluate our proposed method on six representative benchmark datasets, achieving state-of-the-art performance with 1.8-23.7× measured speedup. Furthermore, our method has been deployed to the autonomous racing vehicle of MIT Driverless, achieving larger detection range, higher accuracy and lower latency. [ABSTRACT FROM AUTHOR]

Scene flow represents the 3D motion of every point in the dynamic environments. Like the optical flow that represents the motion of pixels in 2D images, 3D motion representation of scene flow benefits many applications, such as autonomous driving and service robot. This paper studies the problem of scene flow estimation from two consecutive 3D point clouds. In this paper, a novel hierarchical neural network with double attention is proposed for learning the correlation of point features in adjacent frames and refining scene flow from coarse to fine layer by layer. The proposed network has a new more-for-less hierarchical architecture. The more-for-less means that the number of input points is greater than the number of output points for scene flow estimation, which brings more input information and balances the precision and resource consumption. In this hierarchical architecture, scene flow of different levels is generated and supervised respectively. A novel attentive embedding module is introduced to aggregate the features of adjacent points using a double attention method in a patch-to-patch manner. The proper layers for flow embedding and flow supervision are carefully considered in our network designment. Experiments show that the proposed network outperforms the state-of-the-art performance of 3D scene flow estimation on the FlyingThings3D and KITTI Scene Flow 2015 datasets. We also apply the proposed network to the realistic LiDAR odometry task, which is a key problem in autonomous driving. The experiment results demonstrate that our proposed network can outperform the ICP-based method and shows good practical application ability. The source codes will be released on https://github.com/IRMVLab/HALFlow. [ABSTRACT FROM AUTHOR]

Deep object detection models trained on clean images may not generalize well on degraded images due to the well-known domain shift issue. This hinders their application in real-life scenarios such as video surveillance and autonomous driving. Though domain adaptation methods can adapt the detection model from a labeled source domain to an unlabeled target domain, they struggle in dealing with open and compound degradation types. In this paper, we attempt to address this problem in the context of object detection by proposing a robust object Detector via Adversarial Novel Style Exploration (DANSE). Technically, DANSE first disentangles images into domain-irrelevant content representation and domain-specific style representation under an adversarial learning framework. Then, it explores the style space to discover diverse novel degradation styles that are complementary to those of the target domain images by leveraging a novelty regularizer and a diversity regularizer. The clean source domain images are transferred into these discovered styles by using a content-preserving regularizer to ensure realism. These transferred source domain images are combined with the target domain images and used to train a robust degradation-agnostic object detection model via adversarial domain adaptation. Experiments on both synthetic and real benchmark scenarios confirm the superiority of DANSE over state-of-the-art methods. [ABSTRACT FROM AUTHOR]

In recent years, the growing development of Connected Autonomous Vehicles (CAV), Intelligent Transport Systems (ITS), and 5G communication networks have led to the advent of Autonomous Intersection Management (AIM) systems. AIMs present a new paradigm for CAV control in future cities, taking control of CAVs in scenarios where cooperation is necessary and allowing safe and efficient traffic flows, eliminating traffic signals. So far, the development of AIM algorithms has been based on basic control algorithms, without the ability to adapt or keep learning new situations. To solve this, in this paper we present a new advanced AIM approach based on end-to-end Multi-Agent Deep Reinforcement Learning (MADRL) and trained using Curriculum through Self-Play, called advanced Reinforced AIM (adv.RAIM). adv.RAIM enables the control of CAVs at intersections in a collaborative way, autonomously learning complex real-life traffic dynamics. In addition, adv.RAIM provides a new way to build smarter AIMs capable of proactively controlling CAVs in other highly complex scenarios. Results show remarkable improvements when compared to traffic light control techniques (reducing travel time by 59% or reducing time lost due to congestion by 95%), as well as outperforming other recently proposed AIMs (reducing waiting time by 56%), highlighting the advantages of using MADRL. [ABSTRACT FROM AUTHOR]

Autonomous vehicles have been highlighted as a major growth area for future transportation systems and the deployment of large numbers of these vehicles is expected when safety and legal challenges are overcome. To meet the necessary safety standards, effective collision avoidance technologies are required to ensure that the number of accidents are kept to a minimum. As large numbers of autonomous vehicles, operating together on roads, can be regarded as a swarm system, we propose a bio-inspired collision avoidance strategy using virtual pheromones; an approach that has evolved effectively in nature over many millions of years. Previous research using virtual pheromones showed the potential of pheromone-based systems to maneuver a swarm of robots. However, designing an individual controller to maximise the performance of the entire swarm is a major challenge. In this paper, we propose a novel deep reinforcement learning (DRL) based approach that is able to train a controller that introduces collision avoidance behaviour. To accelerate training, we propose a novel sampling strategy called Highlight Experience Replay and integrate it with a Deep Deterministic Policy Gradient algorithm with noise added to the weights and biases of the artificial neural network to improve exploration. To evaluate the performance of the proposed DRL-based controller, we applied it to navigation and collision avoidance tasks in three different traffic scenarios. The experimental results showed that the proposed DRL-based controller outperformed the manually-tuned controller in terms of stability, effectiveness, robustness and ease of tuning process. Furthermore, the proposed Highlight Experience Replay method outperformed than the popular Prioritized Experience Replay sampling strategy by taking 27% of training time average over three stages. [ABSTRACT FROM AUTHOR]

Nowadays, the autonomous driving transportation systems are at the heart of both academic and industry research for the distinguished advantages including increased network capacity, enhanced punctuality, greater flexibility and improved overall safety level. With the responsibility of transporting passengers in a safe, comfortable and efficient way, the decision making method plays a critical position in the autonomous driving of high-speed trains. Focusing on solving the autonomous decision making problem, this paper proposes a novel learning based framework by combining the deep learning technology with the distributed tracking control approach. To cope with the data insufficiency problem in training the deep learning network, a generative adversarial network (GAN) based data argumentation scheme is proposed to generate data samples that have the same distribution with actual data samples, and a hybrid learning network is constructed to predict the speed trajectory from the multi-attribute data with both temporal sequences and static features. Then, based on the model predictive control (MPC) scheme, a distributed tracking control model is formulated to minimize the tracking deviations and balance the performance of punctuality, energy-efficiency and riding comfort. Further, the dual decomposition technique is adopted to deal with the coupling constraints for the safe distance headway such that the separation for the autonomous driving of high-speed trains is achieved. Finally, simulation experiments based on actual scenarios of the Beijing-Shanghai high-speed railway are conducted to illustrate the effectiveness of our methods. [ABSTRACT FROM AUTHOR]

Autonomous pilot is crucial in integrally promoting the autonomy of an unmanned surface vehicle (USV). However, the integration mechanism of decision and control is still unclear within the entire autonomy. In this paper, by organically bridging path planning and tracking, an autonomous pilot framework with waypoints generation, path smoothing and policy guidance of a USV in congested waters is established, for the first time. Incorporating elite and diversity operations into the genetic algorithm (GA), an elite-duplication GA (EGA) strategy is devised to optimally generate sparse waypoints in a constrained space. The B-spline technique is further deployed to make flexibly smooth interpolation facilitating path smoothing supported by optimal sparse-waypoints. Seamlessly bridged by the parametric smooth path, deep reinforcement learning (DRL) technique is resorted to continuously extract in-depth pilotage policies, i.e., mappings from path tracking errors, collision risks and control constraints to continuous control forces/torques. Eventually, the entire spline-bridged EGA-DRL (SED) framework merits autonomous global-pilotage and local-reaction in an organically modular manner. Comprehensive validations and comparisons in various real-world geographies demonstrate the effectiveness and superiority of the proposed SED autonomous pilot framework. [ABSTRACT FROM AUTHOR]

Knowledge transfer is a promising concept to achieve real-time decision-making for autonomous vehicles. This paper constructs a transfer deep reinforcement learning (RL) framework to transform the driving tasks in the intersection environments. The driving missions at the unsignalized intersection are cast into a left turn, right turn, and running straight for automated vehicles. The goal of the autonomous ego vehicle (AEV) is to drive through the intersection situation efficiently and safely. This objective promotes the studied vehicle to increase its speed and avoid crashing other vehicles. The decision-making policy learned from one driving task is transferred through three transfer rules in another driving mission and evaluated. Simulation results reveal that the decision-making strategies related to similar tasks are transferable and have a high success rate. It indicates that the presented control framework could reduce time consumption and realize online implementation. Therefore, the transfer RL concept is helpful for establishing the real-time decision-making policy for autonomous vehicles. [ABSTRACT FROM AUTHOR]

The development of Autonomous Vehicles (AVs) envisions the promising technology of future Intelligent Transportation Systems (ITS). However, the complex road structures and increased vehicles cause traffic congestion and road safety, which eventually leads to horrible accidents. Cooperative driving of AVs, a groundbreaking initiative of vehicle platooning, epitomizes the next wave in vehicular technology through minimizing accident risks, transport times, costs, energy, and fuel consumption. However, the traditional machine learning-based platooning approaches fail to regulate the policy with the dynamic feature of AVs. This paper proposes a hybrid Deep Reinforcement learning and Genetic algorithm for Smart-Platooning (DRG-SP) the AVs. The leverage of the deep reinforcement learning mechanism addresses the computational complexity and accommodates the high dynamic platoon environments. Adopting the Genetic Algorithm in Deep Reinforcement learning overcomes the slow convergence problem and offers long-term performance. The simulation results reveal that the Smart-Platooning effectively forms and maintains the platoons by minimizing traffic congestion and fuel consumption. [ABSTRACT FROM AUTHOR]

This paper presents a latent space reinforcement learning method for interpretable decision-making of autonomous vehicles at highway on-ramps. This method is based on the latent model and the combination model of the hidden Markov model and Gaussian mixture regression (HMM-GMR). It is difficult for the traditional decision-making method to understand the environment because its input is high-dimensional and lacks an understanding of the task. By utilizing the HMM-GMR model, we can obtain the interpretable state providing semantic information and environment understanding. A framework is proposed to unify representation learning with the deep reinforcement learning (DRL) approach, in which the latent model is used to reduce the dimension of interpretable state by extracting underlying task-relevant information. Experimental results are presented and the results show the right balance between driving safety and efficiency in the challenging scenarios of highway on-ramps merging. [ABSTRACT FROM AUTHOR]

Modern computer vision requires processing large amounts of data, both while training the model and/or during inference, once the model is deployed. Scenarios where images are captured and processed in physically separated locations are increasingly common (e.g. autonomous vehicles, cloud computing, smartphones). In addition, many devices suffer from limited resources to store or transmit data (e.g. storage space, channel capacity). In these scenarios, lossy image compression plays a crucial role to effectively increase the number of images collected under such constraints. However, lossy compression entails some undesired degradation of the data that may harm the performance of the downstream analysis task at hand, since important semantic information may be lost in the process. Moreover, we may only have compressed images at training time but are able to use original images at inference time (i.e. test), or vice versa, and in such a case, the downstream model suffers from covariate shift. In this paper, we analyze this phenomenon, with a special focus on vision-based perception for autonomous driving as a paradigmatic scenario. We see that loss of semantic information and covariate shift do indeed exist, resulting in a drop in performance that depends on the compression rate. In order to address the problem, we propose dataset restoration, based on image restoration with generative adversarial networks (GANs). Our method is agnostic to both the particular image compression method and the downstream task; and has the advantage of not adding additional cost to the deployed models, which is particularly important in resource-limited devices. The presented experiments focus on semantic segmentation as a challenging use case, cover a broad range of compression rates and diverse datasets, and show how our method is able to significantly alleviate the negative effects of compression on the downstream visual task. [ABSTRACT FROM AUTHOR]

Recently, inferring lane change intention has received considerable attention. Due to the high nonlinearity and complexity of traffic contexts, traditional methods cannot satisfy the requirements of long-term prediction tasks and lack the ability of capturing nonlinear temporal dependencies. This paper proposes an intention inference model based on Recurrent Neural Networks (RNN), to tackle time series prediction problems. Considering dynamic interaction among surrounding vehicles, our model takes the sequence motion information of surrounding vehicles as inputs and calculates the congestion of different lanes, integrated with vehicle states of the object vehicle. To illustrate the availability of the proposed RNN intention inference model, a motion planning controller considering intention was developed. A Nonlinear Model Predictive Control (NMPC) was established to planning a safe, sub-optimal path for autonomous driving vehicle under collision avoidance constraints. The experiments on the proposed model were conducted, based on two RNN structure Long-Short Term Memory (LSTM) and Generalized Recurrent Unit (GRU), by Tensorflow with NGSIM data. The motion planning controller is modeled and simulated by Carsim with Simulink for some typical scenarios. Subsequently, experimental results demonstrate that RNN achieves best performance, inferring intention with 96% accuracy, compared with other approaches. [ABSTRACT FROM AUTHOR]

This paper investigates computing resource scheduling for real-time applications in autonomous driving, such as localization and obstacle avoidance. In our considered scenario, autonomous vehicles periodically sense the environment, offload sensor data to an edge server for processing, and receive computing results from the server. Due to mobility and computing latency, a vehicle travels some distance in the duration between the instant of offloading its sensor data and the instant of receiving the computing result. Our objective is finding a scheduling scheme for the edge sever to minimize the above traveled distance of vehicles. The approach is to determine the processing order according to individual vehicle mobility and computing capability of the edge server. We formulate a restless multi-arm bandit (RMAB) problem, design a Whittle index based stochastic scheduling scheme, and determine the index using a deep reinforcement learning (DRL) method. The proposed scheduling scheme avoids the time-consuming policy exploration common in DRL scheduling approaches and makes effectual decisions with low complexity. Extensive simulation results demonstrate that the proposed indexed-based scheme can deliver computing results to the vehicles promptly while adapting to time-variant vehicle mobility. [ABSTRACT FROM AUTHOR]

With the development of artificial intelligence(AI), unmanned vehicles can relieve traffic jamming and decrease the risk of traffic accidents, where deep neural networks (DNNs) play an important role and have become one of the most critical technologies. Nevertheless, DNNs are still susceptible to adversarial examples. Even worse, they also show severe performance degradation when the system needs DNNs to learn new knowledge without forgetting the old one. As unmanned vehicles travel on the road, they need to frequently learn new categories and different representations. Learning all data after the new sample arrives will expend a lot of time and space. As a result, it will affect the deployment of artificial intelligence in unmanned scenes. In recent years, it has been observed that incremental learning technology can solve the above challenges. However, previously reported works mainly focused on batch learning. It is not clear how much impact the adversarial attack will have on the deep learning model when performing incremental learning tasks. This issue exposes the hidden safety risks of unmanned driving and increases discuss opportunities. Therefore, we propose an adversarial attack based on incremental learning techniques for unmanned scenes in this paper. Specifically, it can retain information previously learned by the model. At the same time, it can renew the old model to learn new model, thereby continually adding small perturbation to legitimate examples. A couple of experiments on the Pascal VOC 2012 dataset has been conducted, and the experiment results show that the adversarial attack based on incremental learning techniques has a higher attack success rate. Further, it can improve the successful attack rate by 8.43%. [ABSTRACT FROM AUTHOR]

The Internet of Vehicles (IoV) as a promising application of Internet of Things (IoT) has played a significant role in autonomous driving, by connecting intelligent vehicles. Autonomous driving needs to process the mass environmental sensing data in coordination with surrounding vehicles, and makes an accurate driving judgment accordingly. Since the vehicles always have limited computing resources, processing these data in parallel with efficient task scheduling is one of the most important topics. Most current work focuses on formulating special scenarios and service requirements as optimization problems. However, the complicated and dynamic environment of vehicular computing is hard to model, predict and control, making those previous methods unscalable and unable to reflect the real scenario. In this paper, a Multi-task Deep reinforcement learning approach for scalable parallel Task Scheduling (MDTS) is firstly devised. For avoiding the curse of dimensionality when coping with complex parallel computing environments and jobs with diverse properties, we extend the action selection in Deep Reinforcement Learning (DRL) to a multi-task decision, where the output branches of multi- task learning are fine-matched to parallel scheduling tasks. Child tasks of a job are accordingly assigned to distributed nodes without any human knowledge while the resource competition among parallel tasks is leveraged through shared neural network layers. Moreover, we design an appropriate reward function to optimize multiple metrics simultaneously, with emphasis on specific scenarios. Extensive experiments show that the MDTS significantly increases the overall reward compared with least- connection scheduling and particle swarm optimization algorithm from −16.71, −0.67 to 2.93, respectively. [ABSTRACT FROM AUTHOR]

Accurate knowledge of the vehicle states is the foundation of vehicle motion control. However, in real implementations, sensory signals are always corrupted by delays and noises. Network induced time-varying delays and measurement noises can be a hazard in the active safety of over-actuated electric vehicles (EVs). In this paper, a brain-inspired proprioceptive system based on state-of-the-art deep learning and data fusion technique is proposed to solve this problem in autonomous four-wheel actuated EVs. A deep recurrent neural network (RNN) is trained by the noisy and delayed measurement signals to make accurate predictions of the vehicle motion states. Then unscented Kalman predictor, which is the adaption of unscented Kalman filter in time-varying-delay situations, combines the predictions of the RNN and corrupted sensory signals to provide better perceptions of the locomotion. Simulations with a high-fidelity, CarSim, full-vehicle model are carried out to show the effectiveness of our RNN framework and the entire proprioceptive system. [ABSTRACT FROM AUTHOR]

This paper introduces a method to extract driving behaviors from a human expert driver which are applied to an autonomous agent to reproduce proactive driving behaviors. Deep learning techniques were used to extract latent features from the collected data. Extracted features were clustered into behaviors and used to create velocity profiles allowing an autonomous driving agent could drive in a human-like manner. By using proactive driving behaviors, the agent could limit potential sources of discomfort such as jerk and uncomfortable velocities. Additionally, we proposed a method to compare trajectories where not only the geometric similarity is considered, but also velocity, acceleration and jerk. Experimental results in a simulator implemented in ROS show that the autonomous agent built with the driving behaviors was capable of driving similarly to expert human drivers. [ABSTRACT FROM AUTHOR]

Semantic segmentation is a key step in scene understanding for autonomous driving. Although deep learning has significantly improved the segmentation accuracy, current high-quality models such as PSPNet and DeepLabV3 are inefficient given their complex architectures and reliance on multi-scale inputs. Thus, it is difficult to apply them to real-time or practical applications. On the other hand, existing real-time methods cannot yet produce satisfactory results on small objects such as traffic lights, which are imperative to safe autonomous driving. In this paper, we improve the performance of real-time semantic segmentation from two perspectives, methodology and data. Specifically, we propose a real-time segmentation model coined Narrow Deep Network (NDNet) and build a synthetic dataset by inserting additional small objects into the training images. The proposed method achieves 65.7% mean intersection over union (mIoU) on the Cityscapes test set with only 8.4G floating-point operations (FLOPs) on $1024\times 2048$ inputs. Furthermore, by re-training the existing PSPNet and DeepLabV3 models on our synthetic dataset, we obtained an average 2% mIoU improvement on small objects. [ABSTRACT FROM AUTHOR]

Street Scene Parsing (SSP) is a fundamental and important step for autonomous driving and traffic scene understanding. Recently, Fully Convolutional Network (FCN) based methods have delivered expressive performances with the help of large-scale dense-labeling datasets. However, in urban traffic environments, not all the labels contribute equally for making the control decision. Certain labels such as pedestrian, car, bicyclist, road lane or sidewalk would be more important in comparison with labels for vegetation, sky or building. Based on this fact, in this paper we propose a novel deep learning framework, named Residual Atrous Pyramid Network (RAPNet), for importance-aware SSP. More specifically, to incorporate the importance of various object classes, we propose an Importance-Aware Feature Selection (IAFS) mechanism which automatically selects the important features for label predictions. The IAFS can operate in each convolutional block, and the semantic features with different importance are captured in different channels so that they are automatically assigned with corresponding weights. To enhance the labeling coherence, we also propose a Residual Atrous Spatial Pyramid (RASP) module to sequentially aggregate global-to-local context information in a residual refinement manner. Extensive experiments on two public benchmarks have shown that our approach achieves new state-of-the-art performances, and can consistently obtain more accurate results on the semantic classes with high importance levels. [ABSTRACT FROM AUTHOR]

Degraded image semantic segmentation is of great importance in autonomous driving, highway navigation systems, and many other safety-related applications and it was not systematically studied before. In general, image degradations increase the difficulty of semantic segmentation, usually leading to decreased semantic segmentation accuracy. Therefore, performance on the underlying clean images can be treated as an upper bound of degraded image semantic segmentation. While the use of supervised deep learning has substantially improved the state of the art of semantic image segmentation, the gap between the feature distribution learned using the clean images and the feature distribution learned using the degraded images poses a major obstacle in improving the degraded image semantic segmentation performance. The conventional strategies for reducing the gap include: 1) Adding image-restoration based pre-processing modules; 2) Using both clean and the degraded images for training; 3) Fine-tuning the network pre-trained on the clean image. In this paper, we propose a novel Dense-Gram Network to more effectively reduce the gap than the conventional strategies and segment degraded images. Extensive experiments demonstrate that the proposed Dense-Gram Network yields state-of-the-art semantic segmentation performance on degraded images synthesized using PASCAL VOC 2012, SUNRGBD, CamVid, and CityScapes datasets. [ABSTRACT FROM AUTHOR]

Deep reinforcement learning (DRL) has been successfully applied to end-to-end autonomous driving, especially in simulation environments. However, common DRL approaches used in complex autonomous driving scenarios sometimes are unstable or difficult to converge. This paper proposes two approaches to improve the stability of the policy model training with as few manual data as possible. For the first approach, reinforcement learning is combined with imitation learning to train a feature network with a small amount of manual data for parameters initialization. For the second approach, an auxiliary network is added to the reinforcement learning framework, which can leverage the real-time measurement information to deepen the understanding of environment, without any guide of demonstrators. To verify the effectiveness of these two approaches, simulations in image information-based and lidar information-based end-to-end autonomous driving systems are conducted, respectively. These approaches are not only tested in the virtual game world, but also applied in Gazebo, in which we build a 3D world based on the real vehicle model of Ranger XP900 platform, the real 3D obstacle model, and the real motion constraints with inertial characteristics, so as to ensure that the trained end-to-end autonomous driving model is more suitable for the real world. Experimental results show that the performance is increased by over 45% in the virtual game world, and can converge quickly and stably in Gazebo in which previous methods can hardly converge. [ABSTRACT FROM AUTHOR]

With the ongoing development and improvement of deep learning technology, autonomous vehicles (AVs) have made tremendous progress in recent years. Despite its great potential, AV supported by deep learning technology still faces numerous security threats, which prevent AV from being putting into large-scale practice. Aiming at this challenging situation, in this paper, we would like to exploit two attacks against deep learning algorithms in traffic sign recognition system by leveraging particle swarm optimization. Specifically, we first exploit the PAPSO (poisoning attack with particle swarm optimization) which focuses on the training process of the deep learning algorithms in the traffic sign recognition system, i.e., the attacker injects crafted samples into the training dataset, causing a reduction in classification accuracy of the traffic sign recognition system. Then, we also explore the EAPSO (evasion attack with particle swarm optimization) which on the other hand focuses on the interference process of the deep learning algorithms, i.e., the attacker adds some hardly perceptible perturbations to the targeted test sample, leading to a misclassification on it. Extensive experiments are conducted to shed light on the effectiveness of our attacks, and some corresponding defense strategies are also presented. [ABSTRACT FROM AUTHOR]

Deep learning techniques have been widely used in autonomous driving community for the purpose of environment perception. Recently, it starts being adopted for learning end-to-end controllers for complex driving scenarios. However, the complexity and nonlinearity of the network architecture limits its interpretability to understand driving scenarios and judge the importance of certain visual regions in sensory scenes. In this paper, based on the convolutional neural network (CNN), we propose two complementary frameworks to automatically determine the most contributive regions of the input scenes, offering intuitive knowledge of how a trained end-to-end autonomous vehicle controller understands driving scenarios. In the first framework, a feature map-based method is proposed by leveraging current progress in CNN visualization, in which the deconvolution approach recovers the feature maps to extract features that contribute most to understand driving scenes. In the second framework, the importance level of regions is ranked using the error map between the labeled and predicted control inputs generated by occluding different parts of input scenes, thus providing a pixel-wise rank of importance. Test data sets with extracted contributive regions are input to the CNN controller. Then, different CNN controllers trained with the new data sets preprocessed using our proposed frameworks are verified via closed-loop tests. Results show that both the features identified from the first framework and the regions identified from the second framework are of crucial importance to scene understanding for the controller and can significantly affect the performance of CNN controllers. [ABSTRACT FROM AUTHOR]