Skip to Main content Skip to Navigation

Energy-efficient Straggler Mitigation for Big Data Applications on the Clouds

Abstract : Energy consumption is an important concern for large-scale Big Data processing systems, which results in huge monetary cost. Due to the hardware heterogeneity and contentions between concurrent workloads, stragglers (i.e., tasks performing relatively slower than other tasks) can severely increase the job’s execution time and energy consumption. Consequently, straggler mitigation becomes an important technique to improve the performance of large-scale Big Data processing systems. Typically, it consists of two phases: straggler detection and straggler handling. In the detection phase, slow tasks (e.g., tasks with speed or progress below the average) are marked as stragglers. Then, stragglers are handled using the speculative execution technique. With this technique, a copy of the detected straggler is launched in parallel with the straggler with the expectation that it can finish earlier, thus, reduce the straggler’s execution time. Although a large number of studies have been proposed to improve the performance of Big Data applications using speculative execution technique, few of them have studied the energy efficiency of their solutions. Addressing this lack, we conduct an experimental study to fully understand the impact of straggler mitigation techniques on the performance and the energy consumption of Big Data processing systems. We observe that current straggler mitigation techniques are not energy efficient. As a result, this promotes further studies aiming at higher energy efficiency for straggler mitigation. In terms of straggler detection, we introduce a novel framework for comprehensively characterizing and evaluating straggler detection mechanisms. Accordingly, we propose a new energy-driven straggler detection mechanism. This straggler detection mechanism is implemented into Hadoop and is demonstrated to have higher energy efficiency compared to the state-of-the-art mechanisms. In terms of straggler handling, we present a new speculative copy allocation method, which takes into consideration the impact of resource heterogeneity on performance and energy consumption. Finally, an energy efficient straggler handling mechanism is introduced. This mechanism provides more resource availability for launching speculative copies, by adopting a dynamic resource reservation approach. It is demonstrated, via a trace-driven simulation, to bring a high improvement in energy efficiency.
Document type :
Complete list of metadata

Cited literature [131 references]  Display  Hide  Download
Contributor : ABES STAR :  Contact
Submitted on : Monday, March 5, 2018 - 5:56:09 PM
Last modification on : Friday, August 5, 2022 - 2:54:52 PM


Version validated by the jury (STAR)


  • HAL Id : tel-01669469, version 5


Tien-Dat Phan. Energy-efficient Straggler Mitigation for Big Data Applications on the Clouds. Performance [cs.PF]. École normale supérieure de Rennes, 2017. English. ⟨NNT : 2017ENSR0008⟩. ⟨tel-01669469v5⟩



Record views


Files downloads