External Sorting Algorithm: State-of-the-Art and Future Directions

Abstract

Abstract The advent of the era of big data provides new opportunities and more challenges to sorting algorithms. The traditional internal sorting algorithm cannot adapt to the explosive growth of data, and the memory cannot accommodate all the data for sorting, so the external sorting algorithm arises at the historic moment. Because of the different application scenarios, storage devices and improvement strategies, there are many kinds of external sorting algorithms. Traditional main memory architecture based on DRAM faces the problems of capacity, energy consumption and reliability. Emerging nonvolatile memory technologies are non-volatile, high-density, byte-addressable, low-power, so they can replace persistent storage, main memory or storage class memory. Though NVM devices provide new choices to the revolution of traditional memory and storage system, traditional external sorting algorithms cannot achieve its performance. This paper first sorts out the development of external sorting algorithm, and summarizes it into four kinds of external sorting algorithm based on HDD, embedded device, SSD and NVM. In addition, the classical external sorting algorithms based on different storage devices are listed, and our opinions are put forward. Finally, this paper proposes three problems that need to be solved urgently in the future development of external