Fulgor: A fast and compactk-mer index for large-scale matching and color queries

Abstract

AbstractThe problem of sequence identification or matching — determining the subset of references from a given collection that are likely to contain a query nucleotide sequence — is relevant for many important tasks in Computational Biology, such as metagenomics and pan-genome analysis. Due to the complex nature of such analyses and the large scale of the reference collections a resource-efficient solution to this problem is of utmost importance. The reference collection should therefore be pre-processed into anindexfor fast queries. This poses the threefold challenge of designing an index that is efficient to query, has light memory usage, and scales well to large collections.To solve this problem, we describe how recent advancements in associative, order-preserving,k-mer dictionaries can be combined with a compressed inverted index to implement a fast and compactcolored de Bruijngraph data structure. This index takes full advantage of the fact that unitigs in the colored de Bruijn graph aremonochromatic(allk-mers in a unitig have the same set of references of origin, or “color”), leveraging theorder-preservingproperty of its dictionary. In fact,k-mers are kept in unitig order by the dictionary, thereby allowing for the encoding of the map fromk-mers to their inverted lists in as little as 1 +o(1) bits per unitig. Hence, one inverted list per unitig is stored in the index with almost no space/time overhead. By combining this property with simple but effective compression methods for inverted lists, the index achieves very small space.We implement these methods in a tool calledFulgor. Compared toThemisto, the prior state of the art,Fulgorindexes a heterogeneous collection of 30,691 bacterial genomes in 3.8× less space, a collection of 150,000Salmonella entericagenomes in approximately 2 × less space, is at least twice as fast for color queries, and is 2 − 6× faster to construct.2012 ACM Subject ClassificationApplied computing → Bioinformatics