Towards autoscaling of Apache Flink jobs-Reference-Cited by-同舟云学术

Towards autoscaling of Apache Flink jobs

Published:2021-06-01 Issue:1 Volume:13 Page:39-59
ISSN:2066-7760
Container-title:Acta Universitatis Sapientiae, Informatica
language:en
Short-container-title:

Author:

Varga Balázs¹,Balassi Márton²,Kiss Attila³

Affiliation:

1. ELTE Eötvös Loránd University Budapest , Hungary

2. Cloudera , Budapest , Hungary

3. J. Selye University , Komárno , Slovakia

Abstract

Abstract Data stream processing has been gaining attention in the past decade. Apache Flink is an open-source distributed stream processing engine that is able to process a large amount of data in real time with low latency. Computations are distributed among a cluster of nodes. Currently, provisioning the appropriate amount of cloud resources must be done manually ahead of time. A dynamically varying workload may exceed the capacity of the cluster, or leave resources underutilized. In our paper, we describe an architecture that enables the automatic scaling of Flink jobs on Kubernetes based on custom metrics, and describe a simple scaling policy. We also measure the e ects of state size and target parallelism on the duration of the scaling operation, which must be considered when designing an autoscaling policy, so that the Flink job respects a Service Level Agreement.

Publisher

Walter de Gruyter GmbH

Link

https://www.sciendo.com/pdf/10.2478/ausi-2021-0003

Reference21 articles.

1. [1] T. Abdelzaher, Y. Diao, J.L. Hellerstein, C. Lu, X, Zhu, Introduction to control theory and its application to computing systems, in: Performance Modeling and Engineering, Springer US, Boston, MA, 2008, 185–215. ⇒41

2. [2] B. Brazil, Prometheus: Up & Running : Infrastructure and Application Performance Monitoring, O’Reilly Media, Inc., 2018 ⇒44, 47

3. [3] P. Carbone, G. Fóra, S. Ewen, S. Haridi, K. Tzoumas, Lightweight asynchronous snapshots for distributed dataflows, arXiv preprint 1506.08603, 2015 ⇒40, 50

4. [4] P. Carbone, S. Ewen, G. Fóra, S. Haridi, S. Richter, K. Tzoumas, State management in Apache Flink: Consistent stateful distributed stream processing, in Proc. VLDB Endow. 10, 12 (2017) 1718–1729 ⇒40

5. [5] P. Carbone, A. Katsifodimos, S. Ewen, V. Markl, S. Haridi, K. Tzoumas, Apache flink: Stream and batch processing in a single engine, Bulletin of the IEEE Computer Society Technical Committee on Data Engineering 38, 4 (2015) 28–38 ⇒39

Cited by 3 articles. 订阅此论文施引文献订阅此论文施引文献，注册后可以免费订阅5篇论文的施引文献，订阅后可以查看论文全部施引文献

1. Evaluating Stream Processing Autoscalers;Proceedings of the 18th ACM International Conference on Distributed and Event-based Systems;2024-06-24

2. Towards Evaluating Stream Processing Autoscalers;2023 IEEE 39th International Conference on Data Engineering Workshops (ICDEW);2023-04

3. HYAS: Hybrid Autoscaler Agent for Apache Flink;Lecture Notes in Computer Science;2023