What's the Difference? Incremental Processing with Change Queries in Snowflake-Reference-Cited by-同舟云学术

What's the Difference? Incremental Processing with Change Queries in Snowflake

Published:2023-06-13 Issue:2 Volume:1 Page:1-27
ISSN:2836-6573
Container-title:Proceedings of the ACM on Management of Data
language:en
Short-container-title:Proc. ACM Manag. Data

Author:

Akidau Tyler¹^ORCID,Barbier Paul²^ORCID,Cseri Istvan¹^ORCID,Hueske Fabian²^ORCID,Jones Tyler³^ORCID,Lionheart Sasha¹^ORCID,Mills Daniel¹^ORCID,Pauliukevich Dzmitry²^ORCID,Probst Lukas²^ORCID,Semmler Niklas²^ORCID,Sotolongo Dan³^ORCID,Zhang Boyuan¹^ORCID

Affiliation:

1. Snowflake, Bellevue, WA, USA

2. Snowflake, Berlin, Germany

3. Snowflake, San Mateo, CA, USA

Abstract

Incremental algorithms are the heart and soul of stream processing. Low latency results depend on the ability to react to the subset of changes in a dataset over time rather than reprocessing the entirety of a dataset as it evolves. But while the SQL language is well suited for representing streams of changes (via tables) and their application to tables over time (via DML), it entirely lacks a method to query the changes to a table or view in the first place. In this paper, we present CHANGES queries and STREAM objects, Snowflake's primitives for querying and consuming incremental changes to table objects over time. CHANGES queries and STREAMs have been in use within Snowflake for three years, and see broad adoption across our customers. We describe the semantics of these primitives, discuss the implementation challenges, present an analysis of their usage at Snowflake, and contrast with other offerings.

Publisher

Association for Computing Machinery (ACM)

Link

https://dl.acm.org/doi/pdf/10.1145/3589776

Reference65 articles.

1. Aurora: a new model and architecture for data stream management

2. Amit - the situation manager

3. MillWheel

4. Tyler Akidau , Slava Chernyak , and Reuven Lax . 2018. Streaming systems: the what, where, when, and how of large-scale data processing . O'Reilly Media, Inc. Tyler Akidau, Slava Chernyak, and Reuven Lax. 2018. Streaming systems: the what, where, when, and how of large-scale data processing. O'Reilly Media, Inc.

Cited by 4 articles. 订阅此论文施引文献订阅此论文施引文献，注册后可以免费订阅5篇论文的施引文献，订阅后可以查看论文全部施引文献

1. Continuous Data Ingestion and Transformation in Snowflake;Proceedings of the 18th ACM International Conference on Distributed and Event-based Systems;2024-06-24

2. Simplicity and Elegance in Stream Processing: a 5-Year Odyssey;Proceedings of the 18th ACM International Conference on Distributed and Event-based Systems;2024-06-24

3. OpenIVM: a SQL-to-SQL Compiler for Incremental Computations;Companion of the 2024 International Conference on Management of Data;2024-06-09

4. Enhancing Regular Expression Processing through Field-Programmable Gate Array-Based Multi-Character Non-Deterministic Finite Automata;Electronics;2024-04-24