Comparing Spark with MapReduce

Abstract

Glowworm swarm optimization (GSO) is one of the optimization techniques, which need to be parallelized in order to evaluate large problems with high-dimensional function spaces. There are various issues involved in the parallelization of any algorithm such as efficient communication among nodes in a cluster, load balancing, automatic node failure recovery, and scalability of nodes at runtime. In this article, the authors have implemented the GSO algorithm with the Apache Spark framework. Even though we need to address how to distribute the data in the cluster to improve the efficiency of algorithm, the Spark framework is designed in such a way that one does not need to deal with any actual underlying parallelization details. For the experimentation, two multimodal benchmark functions were used to evaluate the Spark-GSO algorithm with various sizes of dimensionality. The authors evaluate the optimization results of the two evaluation functions as well as they will compare the Spark results with the ones obtained using a previously implemented MapReduce-based GSO algorithm.