Affiliation:
1. Duke Univ., Durham, NC
Abstract
Data sets in large applications are often too massive to fit completely inside the computers internal memory. The resulting input/output communication (or I/O) between fast internal memory and slower external memory (such as disks) can be a major performance bottleneck. In this article we survey the state of the art in the design and analysis of external memory (or EM) algorithms and data structures, where the goal is to exploit locality in order to reduce the I/O costs. We consider a variety of EM paradigms for solving batched and online problems efficiently in external memory. For the batched problem of sorting and related problems such as permuting and fast Fourier transform, the key paradigms include distribution and merging. The paradigm of disk striping offers an elegant way to use multiple disks in parallel. For sorting, however, disk striping can be nonoptimal with respect to I/O, so to gain further improvements we discuss distribution and merging techniques for using the disks independently. We also consider useful techniques for batched EM problems involving matrices (such as matrix multiplication and transposition), geometric data (such as finding intersections and constructing convex hulls), and graphs (such as list ranking, connected components, topological sorting, and shortest paths). In the online domain, canonical EM applications include dictionary lookup and range searching. The two important classes of indexed data structures are based upon extendible hashing and B-trees. The paradigms of filtering and bootstrapping provide a convenient means in online data structures to make effective use of the data accessed from disk. We also reexamine some of the above EM problems in slightly different settings, such as when the data items are moving, when the data items are variable-length (e.g., text strings), or when the allocated amount of internal memory can change dynamically. Programming tools and environments are available for simplifying the EM programming task. During the course of the survey, we report on some experiments in the domain of spatial databases using the TPIE system (transparent parallel I/O programming environment). The newly developed EM algorithms and data structures that incorporate the paradigms we discuss are significantly faster than methods currently used in practice.
Publisher
Association for Computing Machinery (ACM)
Subject
General Computer Science,Theoretical Computer Science
Reference264 articles.
1. A B+-tree structure for large quadtrees
2. Lecture Notes in Computer Science (Venice;ABELLO J.
3. Active disks
4. Contemporary Mathematics;AGARWAL P.K.
Cited by
312 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献
1. Object Graph Programming;Proceedings of the 46th IEEE/ACM International Conference on Software Engineering;2024-02-06
2. Turing machines with two-level memory: New computational models for analyzing the input/output complexity;Theoretical Computer Science;2024-02
3. Optimizing Energy Efficiency in Distributed Web Systems;2023 7th International Symposium on Innovative Approaches in Smart Technologies (ISAS);2023-11-23
4. The Backbone of Computing: An Exploration of Data Structures;International Journal of Advanced Research in Science, Communication and Technology;2023-04-12
5. Data Structures for Data-Intensive Applications: Tradeoffs and Design Guidelines;Foundations and Trends® in Databases;2023