Heterogeneous Big Data Parallel Computing Optimization Model using MPI/OpenMP Hybrid and Sensor Networks

Abstract

For the heterogeneous big data parallel computing model, two levels of parallelism between nodes are not considered, resulting in low efficiency of heterogeneous big data parallel computing and bandwidth to send and receive information, high communication overhead, long model running time and small computational volume. In the paper, we propose an optimization model of heterogeneous big data parallel computing based on a hybrid Multi Point Interface (MPI)/Open Multi-Processing (OpenMP) and Sensor Networks. First, the processor characteristics of heterogeneous big data architecture is analyzed, the parallel tasks among processors are divided, collect the heterogeneous big data to be computed and cluster them, and use the processing results as the input items of the model. Then, a parallel load balancing mechanism is established to optimally divide the parallel computing load of heterogeneous big data, and a parallel computing optimization program is written by combining the hybrid programming mode of MPI and OpenMP and using the hybrid MPI/OpenMP, and finally, the parallel computing optimization of heterogeneous big data is realized by optimizing the parallel communication and determining the model parameters. The results show that the proposed model has a communication bandwidth of 510Mbps, a computational volume of 1.16GB, a model runtime of 24s, and an improved network bandwidth utilization of 93%, which can effectively reduce the communication overhead, and improve the efficiency of parallel computing and bandwidth sending and receiving information in sensor networks, and shorten the model running time.