Tyrex: Size-Based Resource Allocation in MapReduce Frameworks

Bogdan Ghit, Dick Epema

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

8 Citations (Scopus)
60 Downloads (Pure)

Abstract

Many large-scale data analytics infrastructures are employed for a wide variety of jobs, ranging from short interactive queries to large data analysis jobs that may take hours or even days to complete. As a consequence, data-processing frameworks like MapReduce may have workloads consisting of jobs with heavy-tailed processing requirements. With such workloads, short jobs may experience slowdowns that are an order of magnitude larger than large jobs do, while the users may expect slowdowns that are more in proportion with the job sizes. To address this problem of large job slowdown variability in MapReduce frameworks, we design a scheduling system called TYREX that is inspired by the well-known TAGS task assignment policy in distributed-server systems. In particular, TYREX partitions the resources of a MapReduce framework, allowing any job running in any partition to read data stored on any machine, imposes runtime limits in the partitions, and successively executes parts of jobs in a work-conserving way in these partitions until they can run to completion. We develop a statistical model for dynamically setting the runtime limits that achieves near optimal job slowdown performance, and we empirically evaluate TYREX on a cluster system with workloads consisting of both synthetic and real-world benchmarks. We find that TYREX cuts in half the job slowdown variability while preserving the median job slowdown when compared to state-of-the-art MapReduce schedulers such as FIFO and FAIR. Furthermore, TYREX reduces the job slowdown at the 95th percentile by more than 50% when compared to FIFO and by 20-40% when compared to FAIR.
Original languageEnglish
Title of host publication Proceedings - 2016 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2016
Place of PublicationLos Alamitos, CA
PublisherIEEE
Pages11-20
Number of pages10
ISBN (Electronic)978-1-5090-2453-7
DOIs
Publication statusPublished - 21 Jul 2016
EventCCGRID 2016: 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing - Cartagena de Indias, Cartagena, Colombia
Duration: 16 May 201619 May 2016

Conference

ConferenceCCGRID 2016
Abbreviated titleCCGRID 2016
Country/TerritoryColombia
CityCartagena
Period16/05/1619/05/16

Keywords

  • Servers
  • Runtime
  • Time factors
  • Delays
  • Computational modeling
  • Resource management
  • Data Analysis

Fingerprint

Dive into the research topics of 'Tyrex: Size-Based Resource Allocation in MapReduce Frameworks'. Together they form a unique fingerprint.

Cite this