Affiliation:
1. Hasso Plattner Institute, University of Potsdam, Potsdam, Germany
2. Federal University of Technology - Paraná, Campo Mourão, Brazil
Abstract
Inclusion dependencies (INDs) are a well-known type of data dependency, specifying that the values of one column are contained in those of another column. INDs can be used for various purposes, such as foreign-key candidate selection or join partner discovery. The traditional notion of INDs is based on clean data, where the dependencies hold without exceptions. Unfortunately, data often contain errors, preventing otherwise valid INDs from being discovered. A typical response to this problem is to relax the dependency definition using a similarity measure to account for minor data errors, such as typos or different formatting. While this relaxation is known for functional dependencies, for inclusion dependencies no such relaxation has been defined.
We formally introduce similarity inclusion dependencies, which relax the inclusion by demanding the existence only of sufficiently similar values. Similarity inclusion dependencies can fulfill traditional IND use cases, such as foreign-key candidate discovery, even in the presence of dirty data. We present Sawfish, the first algorithm to discover all similarity inclusion dependencies in a given dataset efficiently. Our algorithm combines approaches for the discovery of traditional INDs and string similarity joins with a novel sliding-window approach and lazy candidate validation. Our experimental evaluation shows that Sawfish can outperform a baseline by a factor of up to 6.5.
Publisher
Association for Computing Machinery (ACM)
Reference28 articles.
1. Discovering conditional inclusion dependencies
2. Efficiently Detecting Inclusion Dependencies
3. Loreto Bravo , Wenfei Fan , and Shuai Ma . 2007 . Extending dependencies with conditions . In Proceedings of the International Conference on Very Large Databases (VLDB), 243--254 . Loreto Bravo, Wenfei Fan, and Shuai Ma. 2007. Extending dependencies with conditions. In Proceedings of the International Conference on Very Large Databases (VLDB), 243--254.
4. Relaxed Functional Dependencies—A Survey of Approaches
5. KHyperLogLog: Estimating Reidentifiability and Joinability of Large Data at Scale
Cited by
3 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献