Affiliation:
1. Brown University, Providence, RI
2. Aarhus University, Aarhus C, Denmark
Abstract
It is common practice for data scientists to acquire and integrate disparate data sources to achieve higher quality results. But even with a perfectly cleaned and merged data set, two fundamental questions remain: (1) Is the integrated data set complete? and (2) What is the impact of any unknown (i.e., unobserved) data on query results?
In this work, we develop and analyze techniques to estimate the impact of the unknown data (a.k.a.,
unknown unknowns
) on simple aggregate queries. The key idea is that the overlap between different data sources enables us to estimate the number and values of the missing data items. Our main techniques are parameter-free and do not assume prior knowledge about the distribution; we also propose a parametric model that can be used instead when the data sources are imbalanced. Through a series of experiments, we show that estimating the impact of
unknown unknowns
is invaluable to better assess the results of aggregate queries over integrated data sources.
Publisher
Association for Computing Machinery (ACM)
Reference62 articles.
1. Crowds, clouds, and algorithms
2. Estimating the Number of Species: A Review
3. Estimation of the size of a closed population when capture probabilities vary among animals
4. Anne Chao. 1984. Nonparametric estimation of the number of classes in a population. Scand. J. Stat. (1984) 265--270. Anne Chao. 1984. Nonparametric estimation of the number of classes in a population. Scand. J. Stat. (1984) 265--270.
Cited by
5 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献
1. Approximate Query Answering over Open Data;Proceedings of the Workshop on Human-In-the-Loop Data Analytics;2023-06-18
2. Reptile: Aggregation-level Explanations for Hierarchical Data;Proceedings of the 2022 International Conference on Management of Data;2022-06-10
3. Data-driven approaches: Use of digitized operational data in process safety;Methods in Chemical Process Safety;2022
4. Query-Oriented Answer Imputation for Aggregate Queries;Advances in Databases and Information Systems;2019
5. Northstar;Proceedings of the VLDB Endowment;2018-08