What's the Difference? Incremental Processing with Change Queries in Snowflake-Reference-Cited by-同舟云学术

What's the Difference? Incremental Processing with Change Queries in Snowflake

Published:2023-06-13 Issue:2 Volume:1 Page:1-27
ISSN:2836-6573
Container-title:Proceedings of the ACM on Management of Data
language:en
Short-container-title:Proc. ACM Manag. Data

Author:

Akidau Tyler¹^ORCID,Barbier Paul²^ORCID,Cseri Istvan¹^ORCID,Hueske Fabian²^ORCID,Jones Tyler³^ORCID,Lionheart Sasha¹^ORCID,Mills Daniel¹^ORCID,Pauliukevich Dzmitry²^ORCID,Probst Lukas²^ORCID,Semmler Niklas²^ORCID,Sotolongo Dan³^ORCID,Zhang Boyuan¹^ORCID

Affiliation:

1. Snowflake, Bellevue, WA, USA

2. Snowflake, Berlin, Germany

3. Snowflake, San Mateo, CA, USA

Abstract

Incremental algorithms are the heart and soul of stream processing. Low latency results depend on the ability to react to the subset of changes in a dataset over time rather than reprocessing the entirety of a dataset as it evolves. But while the SQL language is well suited for representing streams of changes (via tables) and their application to tables over time (via DML), it entirely lacks a method to query the changes to a table or view in the first place. In this paper, we present CHANGES queries and STREAM objects, Snowflake's primitives for querying and consuming incremental changes to table objects over time. CHANGES queries and STREAMs have been in use within Snowflake for three years, and see broad adoption across our customers. We describe the semantics of these primitives, discuss the implementation challenges, present an analysis of their usage at Snowflake, and contrast with other offerings.

Publisher

Association for Computing Machinery (ACM)

Link

https://dl.acm.org/doi/pdf/10.1145/3589776

Reference65 articles.

1. Aurora: a new model and architecture for data stream management

2. Amit - the situation manager

3. MillWheel

4. Tyler Akidau , Slava Chernyak , and Reuven Lax . 2018. Streaming systems: the what, where, when, and how of large-scale data processing . O'Reilly Media, Inc. Tyler Akidau, Slava Chernyak, and Reuven Lax. 2018. Streaming systems: the what, where, when, and how of large-scale data processing. O'Reilly Media, Inc.