A Multi-Input File Data Symmetry Placement Method Considering Job Execution Frequency for MapReduce Join Operation

Abstract

In recent years, data-parallel computing frameworks such as Hadoop have become increasingly popular among scientists. Data-grouping-aware multiple input file data placement for Hadoop is becoming increasingly popular. However, we note that many data-grouping-aware data placement schemes for multiple input files do not take MapReduce job execution frequency into account. Through the study, such data placement schemes will increase the data transmission between nodes. The starting point of this paper is that if a certain type of MapReduce job has been executed more frequently recently, then it can be assumed that this type of job will also have a higher chance of being executed later. Based on this assumption, we proposed a data-grouping-aware multiple input files data symmetry placement method based on MapReduce jobs execution frequency (DGAMF). Based on the history of MapReduce job executions, this method first creates an inter-block join access correlation model, then divides the correlated blocks into groups according to this model and gives a mathematical model for data placement. The model can be used to guide the placement of data blocks centrally to solve the node load balancing issue caused by data asymmetry. Using the proposed method, correlated blocks from the same groups were placed in the same set of nodes, thereby effectively reducing the amount of data transmitted between nodes. Our data placement method was validated by setting up an experimental Hadoop environment. Experimental results showed that the proposed method effectively processed massive datasets and improved MapReduce’s efficiency significantly.