Parallel Implementation of Swarm Intelligent Algorithms in a Spark-Based Cloud Computing Environment

Abstract

Swarm intelligent algorithms can effectively tackle optimization problems that are difficult to solve by using traditional optimization algorithms. However, with the huge increase in the time and space cost for solving optimization problems, the use of swarm intelligent algorithms suffer from the limitation of overly long computation time. Based on Spark, which is the most popular open-source distributed computing framework, this paper studies specifically using swarm intelligent algorithms to solve combinatorial optimization problems. Based on the characteristics of typical swarm intelligent algorithms, we develop Spark-based parallel implementation of these algorithms to accelerate the population updating and parameter tuning procedures involved in swarm intelligence. Specifically, we first initialize the swarm and generate the initial solution, then perform the distributed iterative evolution procedure, and finally obtain the optimal solution. In addition, in order to improve solution quality, we rely on the Spark platform to perform distributed parameter tuning. The tuning strategy first generates different parameter combinations according to a given parameter list, then execute swarm intelligent algorithms with different parameter combinations in a distributed and parallel manner, and finally determine the optimal parameter combination by comparing the solutions of all algorithms. Experimental results on benchmark datasets show that the distributed algorithms can significantly enhance the computational efficiency without affecting the solution quality.