Bifocal sampling for skew-resistant join size estimation

Abstract

This paper introduces bifocal sampling, a new technique for estimating the size of an equi-join of two relations. Bifocal sampling classifies tuples in each relation into two groups, sparse and dense, based on the number of tuples with the same join value. Distinct estimation procedures are employed that focus on various combinations for joining tuples (e.g., for estimating the number of joining tuples that are dense in both relations). This combination of estimation procedures overcomes some well-known problems in previous schemes, enabling good estimates with no a priori knowledge about the data distribution. The estimate obtained by the bifocal sampling algorithm is proven to lie with high probability within a small constant factor of the actual join size, regardless of the skew, as long as the join size is Ω( n lg n ), for relations consisting of n tuples. The algorithm requires a sample of size at most O (√ n lg n ). By contrast, previous algorithms using a sample of similar size may require the join size to be Ω( n √ n ) to guarantee an accurate estimate. Experimental results support the theoretical claims and show that bifocal sampling is practical and effective.