Affiliation:
1. Google AI, CA, USA and Tel Aviv University, Israel
Abstract
Unaggregated data, in a streamed or distributed form, are prevalent and come from diverse sources such as interactions of users with web services and IP traffic. Data elements have
keys
(cookies, users, queries), and elements with different keys interleave. Analytics on such data typically utilizes statistics expressed as a sum over keys in a specified segment of a function
f
applied to the frequency (the total number of occurrences) of the key. In particular,
Distinct
is the number of active keys in the segment,
Sum
is the sum of their frequencies, and both are special cases of
frequency cap
statistics, which cap the frequency by a parameter
T
.
Random samples can be very effective for quick and efficient estimation of statistics at query time. Ideally, to estimate statistics for a given function
f
, our sample would include a key with frequency
w
with probability roughly proportional to
f
(
w
). The challenge is that while such “gold-standard” samples can be easily computed after aggregating the data (computing the set of key-frequency pairs), this aggregation is costly: It requires structure of size that is proportional to the number of active keys, which can be very large.
We present a sampling framework for unaggregated data that uses a single pass (for streams) or two passes (for distributed data) and structure size proportional to the desired sample size. Our design unifies classic solutions for Distinct and Sum. Specifically, our ℓ-capped samples provide nonnegative unbiased estimates of any monotone non-decreasing frequency statistics and statistical guarantees on quality that are close to gold standard for cap statistics with
T
=Θ (ℓ). Furthermore, our
multi-objective
samples provide these statistical guarantees on quality for all
concave sub-linear
statistics (the nonnegative span of cap functions) while incurring only a logarithmic overhead on sample size.
Publisher
Association for Computing Machinery (ACM)
Subject
Mathematics (miscellaneous)
Cited by
2 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献
1. Sampling Big Ideas in Query Optimization;Proceedings of the 42nd ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems;2023-06-18
2. Truly Perfect Samplers for Data Streams and Sliding Windows;Proceedings of the 41st ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems;2022-06-12