A Fine-Grained Stateful Data Analytics Method Based on Resilient State Table

Abstract

This article describes how stateful data analytic frameworks have emerged to provide fresh and low-latency results for big data processing. At present, it is desired to achieve the fine-grained data model in Spark data processing framework. However, Spark adopts coarse-grained data model in order to facilitate parallelization, it is challenging in dealing with the fine-grained data access in stateful data analytics. In this paper, the authors introduce a fine-grained stateful data component, Resilient State Table (RST), to Spark framework. For filling the gap between the coarse-grained data model in Spark and the fine-grained data access requirements in stateful data analytics, they devise the programming model of RST which interacts with Spark's coarse-grained memory representation seamlessly, and enable users to query/update the state entries in fine granularity with Spark-like programming interfaces. Performance evaluation experiments in various application fields demonstrate that their proposed solution achieves the improvements in latency, fault-tolerance, as well as scalability.