Standard

An Experimental Performance Evaluation of Autoscalers for Complex Workflows. / Ilyushkin, Alexey; Ali-Eldin, Ahmed; Herbst, Nikolas; Bauer, André; Papadopoulos, Alessandro; Epema, Dick; Iosup, Alexandru.

In: ACM Transactions on Modeling and Performance Evaluation of Computing Systems (TOMPECS), Vol. 3, No. 2, 8, 04.2018, p. 1-32.

Research output: Contribution to journalArticleScientificpeer-review

Harvard

Ilyushkin, A, Ali-Eldin, A, Herbst, N, Bauer, A, Papadopoulos, A, Epema, D & Iosup, A 2018, 'An Experimental Performance Evaluation of Autoscalers for Complex Workflows' ACM Transactions on Modeling and Performance Evaluation of Computing Systems (TOMPECS), vol. 3, no. 2, 8, pp. 1-32. https://doi.org/10.1145/3164537

APA

Ilyushkin, A., Ali-Eldin, A., Herbst, N., Bauer, A., Papadopoulos, A., Epema, D., & Iosup, A. (2018). An Experimental Performance Evaluation of Autoscalers for Complex Workflows. ACM Transactions on Modeling and Performance Evaluation of Computing Systems (TOMPECS), 3(2), 1-32. [8]. https://doi.org/10.1145/3164537

Vancouver

Ilyushkin A, Ali-Eldin A, Herbst N, Bauer A, Papadopoulos A, Epema D et al. An Experimental Performance Evaluation of Autoscalers for Complex Workflows. ACM Transactions on Modeling and Performance Evaluation of Computing Systems (TOMPECS). 2018 Apr;3(2):1-32. 8. https://doi.org/10.1145/3164537

Author

Ilyushkin, Alexey ; Ali-Eldin, Ahmed ; Herbst, Nikolas ; Bauer, André ; Papadopoulos, Alessandro ; Epema, Dick ; Iosup, Alexandru. / An Experimental Performance Evaluation of Autoscalers for Complex Workflows. In: ACM Transactions on Modeling and Performance Evaluation of Computing Systems (TOMPECS). 2018 ; Vol. 3, No. 2. pp. 1-32.

BibTeX

@article{5c8a40c7e403420b9c353bf1c58e85f9,
title = "An Experimental Performance Evaluation of Autoscalers for Complex Workflows",
abstract = "Elasticity is one of the main features of cloud computing allowing customers to scale their resources based on the workload. Many autoscalers have been proposed in the past decade to decide on behalf of cloud customers when and how to provision resources to a cloud application based on the workload utilizing cloud elasticity features. However, in prior work, when a new policy is proposed, it is seldom compared to the state-of-the-art, and is often compared only to static provisioning using a predefined quality of service target. This reduces the ability of cloud customers and of cloud operators to choose and deploy an autoscaling policy, as there is seldom enough analysis on the performance of the autoscalers in different operating conditions and with different applications. In our work, we conduct an experimental performance evaluation of autoscaling policies, using as application model workflows, a popular formalism for automating resource management for applications with well-defined yet complex structures. We present a detailed comparative study of general state-of-the-art autoscaling policies, along with two new workflow-specific policies. To understand the performance differences between the seven policies, we conduct various experiments and compare their performance in both pairwise and group comparisons. We report both individual and aggregated metrics. As many workflows have deadline requirements on the tasks, we study the effect of autoscaling on workflow deadlines. Additionally, we look into the effect of autoscaling on the accounted and hourly based charged costs, and we evaluate performance variability caused by the autoscaler selection for each group of workflow sizes. Our results highlight the trade-offs between the suggested policies, how they can impact meeting the deadlines, and how they perform in different operating conditions, thus enabling a better understanding of the current state-of-the-art.",
keywords = "autoscaling, elasticity, scientific workflows, workflow, benchmarking, metrics",
author = "Alexey Ilyushkin and Ahmed Ali-Eldin and Nikolas Herbst and Andr{\'e} Bauer and Alessandro Papadopoulos and Dick Epema and Alexandru Iosup",
year = "2018",
month = "4",
doi = "10.1145/3164537",
language = "English",
volume = "3",
pages = "1--32",
journal = "ACM Transactions on Modeling and Performance Evaluation of Computing Systems (TOMPECS)",
issn = "2376-3639",
number = "2",

}

RIS

TY - JOUR

T1 - An Experimental Performance Evaluation of Autoscalers for Complex Workflows

AU - Ilyushkin, Alexey

AU - Ali-Eldin, Ahmed

AU - Herbst, Nikolas

AU - Bauer, André

AU - Papadopoulos, Alessandro

AU - Epema, Dick

AU - Iosup, Alexandru

PY - 2018/4

Y1 - 2018/4

N2 - Elasticity is one of the main features of cloud computing allowing customers to scale their resources based on the workload. Many autoscalers have been proposed in the past decade to decide on behalf of cloud customers when and how to provision resources to a cloud application based on the workload utilizing cloud elasticity features. However, in prior work, when a new policy is proposed, it is seldom compared to the state-of-the-art, and is often compared only to static provisioning using a predefined quality of service target. This reduces the ability of cloud customers and of cloud operators to choose and deploy an autoscaling policy, as there is seldom enough analysis on the performance of the autoscalers in different operating conditions and with different applications. In our work, we conduct an experimental performance evaluation of autoscaling policies, using as application model workflows, a popular formalism for automating resource management for applications with well-defined yet complex structures. We present a detailed comparative study of general state-of-the-art autoscaling policies, along with two new workflow-specific policies. To understand the performance differences between the seven policies, we conduct various experiments and compare their performance in both pairwise and group comparisons. We report both individual and aggregated metrics. As many workflows have deadline requirements on the tasks, we study the effect of autoscaling on workflow deadlines. Additionally, we look into the effect of autoscaling on the accounted and hourly based charged costs, and we evaluate performance variability caused by the autoscaler selection for each group of workflow sizes. Our results highlight the trade-offs between the suggested policies, how they can impact meeting the deadlines, and how they perform in different operating conditions, thus enabling a better understanding of the current state-of-the-art.

AB - Elasticity is one of the main features of cloud computing allowing customers to scale their resources based on the workload. Many autoscalers have been proposed in the past decade to decide on behalf of cloud customers when and how to provision resources to a cloud application based on the workload utilizing cloud elasticity features. However, in prior work, when a new policy is proposed, it is seldom compared to the state-of-the-art, and is often compared only to static provisioning using a predefined quality of service target. This reduces the ability of cloud customers and of cloud operators to choose and deploy an autoscaling policy, as there is seldom enough analysis on the performance of the autoscalers in different operating conditions and with different applications. In our work, we conduct an experimental performance evaluation of autoscaling policies, using as application model workflows, a popular formalism for automating resource management for applications with well-defined yet complex structures. We present a detailed comparative study of general state-of-the-art autoscaling policies, along with two new workflow-specific policies. To understand the performance differences between the seven policies, we conduct various experiments and compare their performance in both pairwise and group comparisons. We report both individual and aggregated metrics. As many workflows have deadline requirements on the tasks, we study the effect of autoscaling on workflow deadlines. Additionally, we look into the effect of autoscaling on the accounted and hourly based charged costs, and we evaluate performance variability caused by the autoscaler selection for each group of workflow sizes. Our results highlight the trade-offs between the suggested policies, how they can impact meeting the deadlines, and how they perform in different operating conditions, thus enabling a better understanding of the current state-of-the-art.

KW - autoscaling

KW - elasticity

KW - scientific workflows

KW - workflow

KW - benchmarking

KW - metrics

U2 - 10.1145/3164537

DO - 10.1145/3164537

M3 - Article

VL - 3

SP - 1

EP - 32

JO - ACM Transactions on Modeling and Performance Evaluation of Computing Systems (TOMPECS)

T2 - ACM Transactions on Modeling and Performance Evaluation of Computing Systems (TOMPECS)

JF - ACM Transactions on Modeling and Performance Evaluation of Computing Systems (TOMPECS)

SN - 2376-3639

IS - 2

M1 - 8

ER -

ID: 46651921