BtrBlocks: Efficient Columnar Compression for Data Lakes-Reference-Cited by-同舟云学术

BtrBlocks: Efficient Columnar Compression for Data Lakes

Published:2023-06-13 Issue:2 Volume:1 Page:1-26
ISSN:2836-6573
Container-title:Proceedings of the ACM on Management of Data
language:en
Short-container-title:Proc. ACM Manag. Data

Author:

Kuschewski Maximilian¹^ORCID,Sauerwein David²^ORCID,Alhomssi Adnan²^ORCID,Leis Viktor¹^ORCID

Affiliation:

1. Technische Universität München, München, Germany

2. Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany

Abstract

Analytics is moving to the cloud and data is moving into data lakes. These reside on object storage services like S3 and enable seamless data sharing and system interoperability. To support this, many systems build on open storage formats like Apache Parquet. However, these formats are not optimized for remotely-accessed data lakes and today's high-throughput networks. Inefficient decompression makes scans CPU-bound and thus increases query time and cost. With this work we present BtrBlocks, an open columnar storage format designed for data lakes. BtrBlocks uses a set of lightweight encoding schemes, achieving fast and efficient decompression and high compression ratios.

Funder

European Research Council

Publisher

Association for Computing Machinery (ACM)

Link

https://dl.acm.org/doi/pdf/10.1145/3589263

Reference61 articles.

1. October 11 2022. https://github.com/lemire/FastPFor. October 11 2022. https://github.com/lemire/FastPFor.

2. October 11 2022. https://github.com/cwida/fsst. October 11 2022. https://github.com/cwida/fsst.

3. October 14 2022. https://github.com/apache/arrow/blob/883580883aab748fe94336cbed844f09e015178f/cpp/src/parquet/column_writer.cc#L1376. October 14 2022. https://github.com/apache/arrow/blob/883580883aab748fe94336cbed844f09e015178f/cpp/src/parquet/column_writer.cc#L1376.

4. October 14 2022. https://aws.amazon.com/ec2/pricing/on-demand/. October 14 2022. https://aws.amazon.com/ec2/pricing/on-demand/.

5. October 14 2022. https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance-guidelines.html. October 14 2022. https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance-guidelines.html.

Cited by 8 articles. 订阅此论文施引文献订阅此论文施引文献，注册后可以免费订阅5篇论文的施引文献，订阅后可以查看论文全部施引文献

1. Two Birds With One Stone: Designing a Hybrid Cloud Storage Engine for HTAP;Proceedings of the VLDB Endowment;2024-07

2. NULLS!: Revisiting Null Representation in Modern Columnar Formats;Proceedings of the 20th International Workshop on Data Management on New Hardware;2024-06-09

3. Accelerating GPU Data Processing using FastLanes Compression;Proceedings of the 20th International Workshop on Data Management on New Hardware;2024-06-09

4. On Tuning Raft for IoT Workload in Apache IoTDB;2024 IEEE 40th International Conference on Data Engineering (ICDE);2024-05-13

5. AdaEdge: A Dynamic Compression Selection Framework for Resource Constrained Devices;2024 IEEE 40th International Conference on Data Engineering (ICDE);2024-05-13