MorphStream: Adaptive Scheduling for Scalable Transactional Stream Processing on Multicores-Reference-Cited by-同舟云学术

MorphStream: Adaptive Scheduling for Scalable Transactional Stream Processing on Multicores

Published:2023-05-26 Issue:1 Volume:1 Page:1-26
ISSN:2836-6573
Container-title:Proceedings of the ACM on Management of Data
language:en
Short-container-title:Proc. ACM Manag. Data

Author:

Mao Yancan¹^ORCID,Zhao Jianjun²^ORCID,Zhang Shuhao³^ORCID,Liu Haikun²^ORCID,Markl Volker⁴^ORCID

Affiliation:

1. National University of Singapore, Singapore, Singapore

2. Huazhong University of Science and Technology, Wuhan, China

3. Singapore University of Technology and Design, Singapore, Singapore

4. Technische Universität Berlin, Berlin, Germany

Abstract

Transactional stream processing engines (TSPEs) differ significantly in their designs, but all rely on non- adaptive scheduling strategies for processing concurrent state transactions. Subsequently, none exploit multicore parallelism to its full potential due to complex workload dependencies. This paper introduces MorphStream, which adopts a novel approach by decomposing scheduling strategies into three dimensions and then strives to make the right decision along each dimension, based on analyzing the decision trade-offs under varying workload characteristics. Compared to the state-of-the-art, MorphStream achieves up to 3.4 times higher throughput and 69.1% lower processing latency for handling real-world use cases with complex and dynamically changing workload dependencies.

Funder

DFG Priority Program

German Federal Ministry of Education and Research (BMBF) under grants BIFOLD - Berlin Institute for the Foundations of Learning and Data

German Federal Ministry of Education and Research (BMBF) under grants BBDC - Berlin Big Data Center

National Research Foundation, Singapore and Infocomm Media Development Authority under its Future Communications Research & Development Programme

SUTD Start-up Research Grant

National Natural Science Foundation of China

Publisher

Association for Computing Machinery (ACM)

Link

https://dl.acm.org/doi/pdf/10.1145/3588913

Reference41 articles.

1. 2018. Data Artisans Streaming Ledger Serializable ACID Transactions on Streaming Data https://www.data-artisans.com/blog/serializable-acid-transactions-on-streaming-data. (2018). 2018. Data Artisans Streaming Ledger Serializable ACID Transactions on Streaming Data https://www.data-artisans.com/blog/serializable-acid-transactions-on-streaming-data. (2018).

2. (2018). Serializable ACID Transactions on Streaming Data. https://www.ververica.com/blog/serializable-acid-transactions-on-streaming-data (2018). Serializable ACID Transactions on Streaming Data. https://www.ververica.com/blog/serializable-acid-transactions-on-streaming-data

3. FlowDB

4. TSpoon: Transactions on a stream processor

5. The CQL continuous query language: semantic foundations and query execution

Cited by 4 articles. 订阅此论文施引文献订阅此论文施引文献，注册后可以免费订阅5篇论文的施引文献，订阅后可以查看论文全部施引文献

1. MorphStream: Scalable Processing of Transactions over Streams;2024 IEEE 40th International Conference on Data Engineering (ICDE);2024-05-13

2. Fast Parallel Recovery for Transactional Stream Processing on Multicores;2024 IEEE 40th International Conference on Data Engineering (ICDE);2024-05-13

3. General-purpose data stream processing on heterogeneous architectures with WindFlow;Journal of Parallel and Distributed Computing;2024-02

4. A survey on transactional stream processing;The VLDB Journal;2023-09-27