Search

Hadoop is a popular computing framework designed to deliver timely and cost-effective data processing on a large cluster of commodity machines. It relieves the burden of the programmers dealing with distributed programming, and an ecosystem of Big Data solutions has developed around it. However, Hadoop's job execution time can greatly depend on its runtime configurations and resource selections. Given the more than 100 job configuration settings provided by Hadoop, and diverse resource instance options in a cloud or virtualized computing environment, running Hadoop jobs still requires a substantial amount of expertise and experience. To address this challenge, we apply a deep neural network to predict Hadoop's job time based on historical execution data, and propose optimization methods to reduce job execution time and cost. The results show that our prediction method achieves almost 90% time prediction accuracy and clearly outperforms three other state-of-the-art regression-based prediction methods. Based on the time prediction, our proposed configuration search method and job scheduling algorithm successfully shorten the execution time of a single Hadoop job by more than a factor of 2 and reduce the time of processing a batch of Hadoop jobs by 40%∼65%. • We aim to optimize the performance and cost of running Hadoop MapReduce workload in a cloud environment with on-demand resources. • We propose a deep learning approach to predict the execution time of MapReduce jobs under any given resource allocation and Hadoop configuration setting. • We proposed a 2-step approximate search algorithm for finding the optimal resource allocation and execution configuration of individual job, and a 2D variable size bin packing algorithm for maximize overall resource utilization. • The experiment results show that our prediction method achieved almost 90% of time prediction accuracy and clearly out-performed three other state-of-art machine learning prediction methods. • Our optimization strategies successfully shorten the execution time of a single Hadoop job by more than a factor of 2 and reduce the time of processing a batch of Hadoop jobs by 40%∼65%. [ABSTRACT FROM AUTHOR]

Distributed training is needed to shorten the training time of deep neural networks. However, the communication overhead often hurts performance efficiency, especially in a distributed computing environment with limited network bandwidth. Hence, gradient compression techniques have been proposed to reduce communication time. But, compression also has the risk of causing lower model accuracy and longer training time due to compression loss and compression time. As a result, compression may not consistently achieve desired results, and there are limited discussions on when and which compression should be used. To address this problem, we propose a performance-driven hybrid compression solution. We make three main contributions. (1) We describe a hybrid compression strategy that chooses the compression method for individual model gradients. (2) We build an offline performance estimator and an online loss monitor to ensure the compression decision can minimize training time without sacrificing mode accuracy. (3) Our implementation can be imported to existing deep learning frameworks and applicable to a wide range of compression methods. Up to 3.6x training performance speedup was observed compared to other state-of-the-art methods. • Our goal is to constrict a better compression strategy for training a DNN model in distributed computing environment. • Our proposed compression strategy is fine-grained, hybrid and performance-driven. • Our method is built based on an offline performance estimator, an online loss monitor, and a linear search time optimization algorithm. • Our approach is implemented as a lightweight library that can be imported to existing deep learning frameworks. • Our evaluations show that we achieved up to 3.6x training performance speedup compared to other compression methods. [ABSTRACT FROM AUTHOR]

Recently, the deep learning schemes have been well investigated for improving the driving safety and efficiency in the transportation systems. In this paper, a probabilistic model named as CPGN (Collision Prediction model based on GA-optimized Neural Network) for decision-making in the rear-end collision avoidance system is proposed, targeting modeling the impact of important influential factors of collisions on the occurring probability of possible accidents in the Internet of Vehicles (IoV). The decision on how to serve the chauffeur is determined by a typical deep learning model, i.e., the BP neural network through evaluating the possible collision risk with V2I (Vehicle-to-Infrastructure) communication, V2V (Vehicle-to-Vehicle) communication and GPS infrastructure supporting. The proper structure of our BP neural network model is deeply learned with training data generated from VISSIM with multiple influential factors considered. In addition, since the selection of the connection coefficient array and thresholds of the neural network has great randomness, a local optimization issue is readily occurring during the modeling procedure. To overcome this problem and consider the ability to efficiently find out a global optimization, this paper chooses the genetic algorithm to optimize the coefficient array and thresholds of proposed neural network. For the purpose of enhancing the convergence speed of the proposed model, we further adjust the studying rate according to the relationship between the actual and predicated values of two adjacent iterations. Simulation results demonstrate that the proposed collision risk evaluation framework could offer rationale estimations to the possible collision risk in car-following scenarios for the next discrete monitoring interval. [ABSTRACT FROM AUTHOR]