Research on dynamic load balancing of data flow under big data platform

Abstract

In the big data platform, because of the large amount of data, the problem of load imbalance is prominent. Most of the current load balancing methods have problems such as high data flow loss rate and long response time; therefore, more effective load balancing method is urgently needed. Taking HBase as the research subject, the study analyzed the dynamic load balancing method of data flow. First, the HBase platform was introduced briefly, and then the dynamic load-balancing algorithm was designed. The data flow was divided into blocks, and then the load of nodes was predicted based on the grey prediction GM(1,1) model. Finally, the load was migrated through the dynamic adjustable method to achieve load balancing. The experimental results showed that the accuracy of the method for load prediction was high, the average error percentage was 0.93%, and the average response time was short; under 3000 tasks, the response time of the method designed in this study was 14.17% shorter than that of the method combining TV white space (TVWS) and long-term evolution (LTE); the average flow of nodes with the largest load was also smaller, and the data flow loss rate was basically 0%. The experimental results show the effectiveness of the proposed method, which can be further promoted and applied in practice.