Affiliation:
1. University of Edinburgh
2. Alibaba Group
Abstract
This paper studies how to catch duplicates, mismatches and conflicts in the same process. We adopt a class of entity enhancing rules that embed machine learning predicates, unify entity resolution and conflict resolution, and are collectively defined across multiple relations. We detect discrepancies as violations of such rules. We establish the complexity of discrepancy detection and incremental detection problems with the rules; they are both NP-complete and W[1]-hard. To cope with the intractability and scale with large datasets, we develop parallel algorithms and parallel incremental algorithms for discrepancy detection. We show that both algorithms are parallelly scalable,
i.e.
, they guarantee to reduce runtime when more processors are used. Moreover, the parallel incremental algorithm is relatively bounded. The complexity bounds and algorithms carry over to denial constraints, a special case of the entity enhancing rules. Using real-life and synthetic datasets, we experimentally verify the effectiveness, scalability and efficiency of the algorithms.
Subject
General Earth and Planetary Sciences,Water Science and Technology,Geography, Planning and Development
Cited by
8 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献
1. Splitting Tuples of Mismatched Entities;Proceedings of the ACM on Management of Data;2023-12-08
2. Discovering Top-k Rules using Subjective and Objective Criteria;Proceedings of the ACM on Management of Data;2023-05-26
3. Making It Tractable to Catch Duplicates and Conflicts in Graphs;Proceedings of the ACM on Management of Data;2023-05-26
4. Learning and Deducing Temporal Orders;Proceedings of the VLDB Endowment;2023-04
5. Big graphs;Proceedings of the VLDB Endowment;2022-08