Search

With the fast growth of big data applications, it has brought about a huge increase in the energy consumption for big data processing in Cloud data centers. In this study, a frequency-aware and energy-saving strategy based on dynamic voltage and frequency scaling (abbreviated as FAESS-DVFS) is proposed to reduce energy consumption for big data processing in Spark on YARN. Energy saving in two layers (YARN layer and Spark layer) has been designed and implemented for the proposed method. First, an optimal CPU frequency is presented in YARN layer based on the minimum energy efficiency ratio (EER) which can be obtained from status monitoring module. Then, a task scheduling method in Spark layer is constructed to optimize the energy consumption by dynamically adjusting the CPU frequency of nodes in the life cycle of different stages. Test on Hibench, the proposed method can achieve substantial energy saving of up to 29.5% for big data processing compared with the default algorithm in Spark on YARN while satisfying SLA constrains. [ABSTRACT FROM AUTHOR]

In virtualized data centers, consolidation of virtual machines (VMs) on minimizing the number of total physical machines (PMs) has been recognized as a very efficient approach. This paper considers the energy-efficient consolidation of VMs in a cloud datacenter. Concentrating on CPU-intensive applications, the objective is to schedule all requests non-preemptively, subjecting to constraints of PM capacities and running time interval spans, to make the total energy consumption of all PMs is minimized (called MinTE for abbreviation). The MinTE problem is NP-complete in general. We propose a self-adaptive approach called SAVE. The approach makes decisions of the assignment and migration of VMs by probabilistic processes and is based exclusively on local information. Both simulation and real environment test show that our proposed method SAVE can reduce energy consumption about 30 % against VMWare DRS and 10–20% against ecoCloud on average. Extensive experiments show that our method outperforms the existing method and achieves significant energy savings and high utilization. [ABSTRACT FROM AUTHOR]

Large-scale MapReduce clusters that routinely process big data bring challenges to the cloud computing. One of the key challenges is to reduce the response time of these MapReduce clusters by minimizing their makespans. It is observed that the order in which these jobs are executed can have a significant impact on their overall makespans and resource utilization. In this work, we consider a scheduling model for multiple MapReduce jobs. The goal is to design a job scheduler that minimizes the makespan of such a set of MapReduce jobs. We exploit classical Johnson model and propose a novel framework HScheduler, which combines features of both classical Johnson's algorithm and MapReduce to minimize the makespan for both offline and online jobs. Our Offline HScheduler reaches the theoretical lower bound (optimum) and Online HScheduler is 2-competitive which is the best-known constant ratio for minimizing the makespan. Through extensive real data tests, we find that HScheduler has better performance than the best-known approach by 10.6-11.7 % on average for offline scheduling and 8-10 % on average for online scheduling. The HScheduler can be applied to improve responsive time, throughput and energy efficiency in cloud computing. [ABSTRACT FROM AUTHOR]

This paper considers online energy-efficient scheduling of virtual machines (VMs) for Cloud data centers. Each request is associated with a start-time, an end-time, a processing time and a capacity demand from a Physical Machine (PM). The goal is to schedule all of the requests non-preemptively in their start-time-end-time windows, subjecting to PM capacity constraints, such that the total busy time of all used PMs is minimized (called MinTBT-ON for abbreviation). This problem is a fundamental scheduling problem for parallel jobs allocation on multiple machines; it has important applications in power-aware scheduling in cloud computing, optical network design, customer service systems, and other related areas. Offline scheduling to minimize busy time is NP-hard already in the special case where all jobs have the same processing time and can be scheduled in a fixed time interval. One best-known result for MinTBT-ON problem is a g-competitive algorithm for general instances and unit-size jobs using First-Fit algorithm where g is the total capacity of a machine. In this paper, a $(1+\frac{g-2}{k}-\frac{g-1}{k^{2}})$ -competitive algorithm, Dynamic Bipartition-First-Fit (BFF) is proposed and proved for general case, where k is the ratio of the length of the longest interval over the length of the second longest interval for k>1 and g≥2. More results in general and special cases are obtained to improve the best-known bounds. [ABSTRACT FROM AUTHOR]