Load Balancing Scheme for Effectively Supporting Distributed In-Memory Based Computing

Abstract

As digital data have increased exponentially due to an increasing number of information channels that create and distribute the data, distributed in-memory systems were introduced to process big data in real-time. However, when the load is concentrated on a specific node in a distributed in-memory environment, the data access performance is degraded, resulting in an overall degradation in the processing performance. In this paper, we propose a new load balancing scheme that performs data migration or replication according to the loading status in heterogeneous distributed in-memory environments. The proposed scheme replicates hot data when the hot data occurs on the node where a load occurs. If the load of the node increases in the absence of hot data, the data is migrated through a hash space adjustment. In addition, when nodes are added or removed, data distribution is performed by adjusting the hash space with the adjacent nodes. The clients store the metadata of the hot data and reduce the access of the load balancer through periodic synchronization. It is confirmed through various performance evaluations that the proposed load balancing scheme improves the overall load balancing performance.