DRIIS: MapReduce Parameter Optimization of Hadoop Using Genetic Algorithm

Abstract

In this era of massive knowledge, Hadoop, jointly of the foremost unremarkably used massive processing platforms, features a variety of parameters that are closely associated with resource utilization, particularly mainframe and memory. Calibration of these parameters through improvement may increase Hadoop’s resource utilization. Manual calibration of those parameters is just about not possible, thanks to the time price. Within the massive knowledge business, there’s a requirement to mechanically set up parameters and thereby maximize resource usage. The previous automatic calibration strategies take an extended time to realize the optimum configuration, reducing the cluster’s overall performance. With the help of a novel perceptive procedure, each genetic programming and a genetic rule are supported with a view to enhancing the performance of a Hadoop MapReduce work, we propose an assistant in nursing the best configuration finder. We have a tendency to use the algorithms on top to search out the simplest values for parameter settings. Experiments were performed on four common applications, WordCount, TeraSort, Index and Grep, and eight virtual machines (VMs) in a very typical Hadoop cluster. Our projected methodology will increase MapReduce job potency by 53.63% for a one GB dataset and by 67.4% for a five GB dataset, in keeping with the findings, and by 73.68% on a 10 GB dataset; additionally, for a TeraSort programmed application, MapReduce job potency will increase by 52.62% for an one GB dataset, 61.2% for a five GB dataset, and 55.17% for a 10[Formula: see text]GB dataset. MapReduce jobs boost the performance of Grep applications by 44.4% for a one GB dataset, 56.25% for a five GB dataset, and 49.44% for a 10 GB dataset.