An incrementally updatable and scalable system for large-scale sequence search using the Bentley–Saxe transformation
Author:
Almodaresi Fatemeh1ORCID,
Khan Jamshed1ORCID,
Madaminov Sergey2,
Ferdman Michael2,
Johnson Rob3,
Pandey Prashant3,
Patro Rob1ORCID
Affiliation:
1. Department of Computer Science, University of Maryland , USA
2. Department of Computer Science, Stony Brook University , USA
3. VMware Research, Palo Alto, CA 94301, USA
Abstract
Abstract
Motivation
In the past few years, researchers have proposed numerous indexing schemes for searching large datasets of raw sequencing experiments. Most of these proposed indexes are approximate (i.e. with one-sided errors) in order to save space. Recently, researchers have published exact indexes—Mantis, VariMerge and Bifrost—that can serve as colored de Bruijn graph representations in addition to serving as k-mer indexes. This new type of index is promising because it has the potential to support more complex analyses than simple searches. However, in order to be useful as indexes for large and growing repositories of raw sequencing data, they must scale to thousands of experiments and support efficient insertion of new data.
Results
In this paper, we show how to build a scalable and updatable exact raw sequence-search index. Specifically, we extend Mantis using the Bentley–Saxe transformation to support efficient updates, called Dynamic Mantis. We demonstrate Dynamic Mantis’s scalability by constructing an index of ≈40K samples from SRA by adding samples one at a time to an initial index of 10K samples. Compared to VariMerge and Bifrost, Dynamic Mantis is more efficient in terms of index-construction time and memory, query time and memory and index size. In our benchmarks, VariMerge and Bifrost scaled to only 5K and 80 samples, respectively, while Dynamic Mantis scaled to more than 39K samples. Queries were over 24× faster in Mantis than in Bifrost (VariMerge does not immediately support general search queries we require). Dynamic Mantis indexes were about 2.5× smaller than Bifrost’s indexes and about half as big as VariMerge’s indexes.
Availability and implementation
Dynamic Mantis implementation is available at https://github.com/splatlab/mantis/tree/mergeMSTs.
Supplementary information
Supplementary data are available at Bioinformatics online.
Funder
NIH
NSF
Applied Mathematics Program of the DOE Office of Advanced Scientific Computing Research
Exascale Computing Project
U.S. Department of Energy Office of Science and the National Nuclear Security Administration
NSF CISE Research Infrastructure
Publisher
Oxford University Press (OUP)
Subject
Computational Mathematics,Computational Theory and Mathematics,Computer Science Applications,Molecular Biology,Biochemistry,Statistics and Probability
Reference31 articles.
1. An efficient, scalable, and exact representation of high-dimensional color information enabled using de Bruijn graph search;Almodaresi;J. Comput. Biol,2020
2. Decomposable searching problems I. Static-to-dynamic transformation;Bentley;J. Algor,1980
Cited by
1 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献