Due to the diversity in the applications that run in clusters, many different application frameworks have been developed, such as MapReduce for data-intensive batch jobs and Spark for interactive data analytics. A framework is first deployed in a cluster, and then starts executing a large set of jobs that are submitted over time. When multiple such frameworks with time-varying resource demands are presentin a single cluster, static allocation of resources on a per-framework basis leads to low system utilization and resource fragmentation. In this paper, we present koala-f, a resource manager that dynamically provides resources to frameworks by employing a feedback loop to collecttheir possibly different performance metrics. Frameworks periodically -- not necessarily with the same frequency -- report the values of their performancemetrics to koala-f, which then either rebalances their resources individuallyagainst the idle-resource pool, or, when the latter is empty, rebalances their resources amongst them. We demonstrate the effectiveness of koala-f with experiments in a real system.
Original languageEnglish
Title of host publication16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2016
Place of PublicationLos Alamitos, CA
PublisherIEEE Computer Society
Number of pages10
ISBN (Electronic)978-1-5090-2453-7
Publication statusPublished - 2016
Externally publishedYes
EventCCGRID 2016: 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing - Cartagena de Indias, Cartagena, Colombia
Duration: 16 May 201619 May 2016


ConferenceCCGRID 2016
Abbreviated titleCCGRID 2016

    Research areas

  • Measurement, Resource management, Color, Schedules, Watermarking, Dynamic scheduling, Sparks

ID: 29617618