Abstract
We present new hash tables for joins, and a hash join based on them, that consumes far less memory and is usually faster than recently published in-memory joins. Our hash join is not restricted to outer tables that fit wholly in memory. Key to this hash join is a new concise hash table (CHT), a linear probing hash table that has 100% fill factor, and uses a sparse bitmap with embedded population counts to almost entirely avoid collisions. This bitmap also serves as a Bloom filter for use in multi-table joins.
We study the random access characteristics of hash joins, and renew the case for non-partitioned hash joins. We introduce a variant of partitioned joins in which only the build is partitioned, but the probe is not, as this is more efficient for large outer tables than traditional partitioned joins. This also avoids partitioning costs during the probe, while at the same time allowing parallel build without latching overheads. Additionally, we present a variant of CHT, called a concise array table (CAT), that can be used when the key domain is moderately dense. CAT is collision-free and avoids storing join keys in the hash table.
We perform a detailed comparison of CHT and CAT against leading in-memory hash joins. Our experiments show that we can reduce the memory usage by one to three orders of magnitude, while also being competitive in performance.
Subject
General Earth and Planetary Sciences,Water Science and Technology,Geography, Planning and Development
Cited by
64 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献
1. CPU and GPU Hash Joins on Skewed Data;2024 IEEE 40th International Conference on Data Engineering Workshops (ICDEW);2024-05-13
2. SQL queries over encrypted databases: a survey;Connection Science;2024-03-05
3. Workload Prediction for Edge Computing;Proceedings of the 25th International Conference on Distributed Computing and Networking;2024-01-04
4. NOCAP: Near-Optimal Correlation-Aware Partitioning Joins;Proceedings of the ACM on Management of Data;2023-12-08
5. Cache-Efficient Top-k Aggregation over High Cardinality Large Datasets;Proceedings of the VLDB Endowment;2023-12