Abstract
Many large-scale data analytics infrastructures are employed for a wide variety of jobs, ranging from short interactive queries to large data analysis jobs that may take hours or even days to complete. As a consequence, data-processing frameworks like MapReduce may have workloads consisting of jobs with heavy-tailed processing requirements. With such workloads, short jobs may experience slowdowns that are an order of magnitude larger than large jobs do, while the users may expect slowdowns that are more in proportion with the job sizes. To address this problem of large job slowdown variability in MapReduce frameworks, we design a scheduling system called TYREX that is inspired by the well-known TAGS task assignment policy in distributed-server systems. In particular, TYREX partitions the resources of a MapReduce framework, allowing any job running in any partition to read data stored on any machine, imposes runtime limits in the partitions, and successively executes parts of jobs in a work-conserving way in these partitions until they can run to completion. We develop a statistical model for dynamically setting the runtime limits that achieves near optimal job slowdown performance, and we empirically evaluate TYREX on a cluster system with workloads consisting of both synthetic and real-world benchmarks. We find that TYREX cuts in half the job slowdown variability while preserving the median job slowdown when compared to state-of-the-art MapReduce schedulers such as FIFO and FAIR. Furthermore, TYREX reduces the job slowdown at the 95th percentile by more than 50% when compared to FIFO and by 20-40% when compared to FAIR.
Original language | English |
---|---|
Title of host publication | Proceedings - 2016 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2016 |
Place of Publication | Los Alamitos, CA |
Publisher | IEEE |
Pages | 11-20 |
Number of pages | 10 |
ISBN (Electronic) | 978-1-5090-2453-7 |
DOIs | |
Publication status | Published - 21 Jul 2016 |
Event | CCGRID 2016: 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing - Cartagena de Indias, Cartagena, Colombia Duration: 16 May 2016 → 19 May 2016 |
Conference
Conference | CCGRID 2016 |
---|---|
Abbreviated title | CCGRID 2016 |
Country/Territory | Colombia |
City | Cartagena |
Period | 16/05/16 → 19/05/16 |
Keywords
- Servers
- Runtime
- Time factors
- Delays
- Computational modeling
- Resource management
- Data Analysis