Affiliation:
1. University of Illinois at Urbana-Champaign
2. LinkedIn Corp
Abstract
Distributed stream processing systems need to support
stateful processing, recover quickly from failures
to resume such processing, and
reprocess
an entire data stream quickly. We present Apache Samza, a distributed system for stateful and fault-tolerant stream processing. Samza utilizes a partitioned local state along with a low-overhead background changelog mechanism, allowing it to scale to massive state sizes (hundreds of TB) per application. Recovery from failures is sped up by re-scheduling based on Host Affinity. In addition to processing infinite streams of events, Samza supports processing a finite dataset as a stream, from either a streaming source (e.g., Kafka), a database snapshot (e.g., Databus), or a file system (e.g. HDFS), without having to change the application code (unlike the popular Lambda-based architectures which necessitate maintenance of separate code bases for batch and stream path processing).
Samza is currently in use at LinkedIn by hundreds of production applications with more than 10, 000 containers. Samza is an open-source Apache project adopted by many top-tier companies (e.g., LinkedIn, Uber, Netflix, TripAdvisor, etc.). Our experiments show that Samza: a) handles state efficiently, improving latency and throughput by more than 100X compared to using a remote storage; b) provides recovery time independent of state size; c) scales performance linearly with number of containers; and d) supports reprocessing of the data stream quickly and with minimal interference on real-time traffic.
Subject
General Earth and Planetary Sciences,Water Science and Technology,Geography, Planning and Development
Cited by
184 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献
1. Benchmarking scalability of stream processing frameworks deployed as microservices in the cloud;Journal of Systems and Software;2024-02
2. AQUA: A Framework for Spatiotemporal Analysis and Visualizations of Water Quality Data at Scale;2023 IEEE International Conference on Big Data (BigData);2023-12-15
3. Practical Storage-Compute Elasticity for Stream Data Processing;Proceedings of the 24th International Middleware Conference: Industrial Track;2023-12-11
4. Pravega;Proceedings of the 24th International Middleware Conference on ZZZ;2023-11-27
5. A survey on the evolution of stream processing systems;The VLDB Journal;2023-11-22