Search

Hadoop is a popular computing framework designed to deliver timely and cost-effective data processing on a large cluster of commodity machines. It relieves the burden of the programmers dealing with distributed programming, and an ecosystem of Big Data solutions has developed around it. However, Hadoop's job execution time can greatly depend on its runtime configurations and resource selections. Given the more than 100 job configuration settings provided by Hadoop, and diverse resource instance options in a cloud or virtualized computing environment, running Hadoop jobs still requires a substantial amount of expertise and experience. To address this challenge, we apply a deep neural network to predict Hadoop's job time based on historical execution data, and propose optimization methods to reduce job execution time and cost. The results show that our prediction method achieves almost 90% time prediction accuracy and clearly outperforms three other state-of-the-art regression-based prediction methods. Based on the time prediction, our proposed configuration search method and job scheduling algorithm successfully shorten the execution time of a single Hadoop job by more than a factor of 2 and reduce the time of processing a batch of Hadoop jobs by 40%∼65%. • We aim to optimize the performance and cost of running Hadoop MapReduce workload in a cloud environment with on-demand resources. • We propose a deep learning approach to predict the execution time of MapReduce jobs under any given resource allocation and Hadoop configuration setting. • We proposed a 2-step approximate search algorithm for finding the optimal resource allocation and execution configuration of individual job, and a 2D variable size bin packing algorithm for maximize overall resource utilization. • The experiment results show that our prediction method achieved almost 90% of time prediction accuracy and clearly out-performed three other state-of-art machine learning prediction methods. • Our optimization strategies successfully shorten the execution time of a single Hadoop job by more than a factor of 2 and reduce the time of processing a batch of Hadoop jobs by 40%∼65%. [ABSTRACT FROM AUTHOR]

Distributed training is needed to shorten the training time of deep neural networks. However, the communication overhead often hurts performance efficiency, especially in a distributed computing environment with limited network bandwidth. Hence, gradient compression techniques have been proposed to reduce communication time. But, compression also has the risk of causing lower model accuracy and longer training time due to compression loss and compression time. As a result, compression may not consistently achieve desired results, and there are limited discussions on when and which compression should be used. To address this problem, we propose a performance-driven hybrid compression solution. We make three main contributions. (1) We describe a hybrid compression strategy that chooses the compression method for individual model gradients. (2) We build an offline performance estimator and an online loss monitor to ensure the compression decision can minimize training time without sacrificing mode accuracy. (3) Our implementation can be imported to existing deep learning frameworks and applicable to a wide range of compression methods. Up to 3.6x training performance speedup was observed compared to other state-of-the-art methods. • Our goal is to constrict a better compression strategy for training a DNN model in distributed computing environment. • Our proposed compression strategy is fine-grained, hybrid and performance-driven. • Our method is built based on an offline performance estimator, an online loss monitor, and a linear search time optimization algorithm. • Our approach is implemented as a lightweight library that can be imported to existing deep learning frameworks. • Our evaluations show that we achieved up to 3.6x training performance speedup compared to other compression methods. [ABSTRACT FROM AUTHOR]