The Research of Spark Memory Optimization Based on Non-Volatile Memory

Abstract

With the advent of the significant data era, more and more data information needs to be processed, bringing tremendous challenges to storage and computing. The spark amount of data is getting larger and larger, and the I/O bottleneck of computing and scheduling from the disk has increasingly become an essential factor restricting performance. The spark came into being and proposed in-memory computing, which significantly improved the computing speed. In addition, the high rate of the memory is easy to lose without power, and the small but expensive feature is also an urgent need to improve. The emergence of new non-volatile memory (NVM) not only brings the characteristics of non-volatile, large capacity, low latency but also brings new opportunities and challenges to the storage system. Therefore, based on the emergence of NVM and the problems to be improved in Spark memory, this paper proposes an NVM-based Spark memory optimization method. Add NVM to the Spark memory system, build a hybrid storage structure of NVM and memory, and make the partition management for NVM storage. What’s more, add some new persistence levels and optimize RDDs and other vital data. In the end, make the related optimization for cache and recovery.