Novel Big Data Networking Framework Using Multihoming Optimization for Distributed Stream Computing

Abstract

One of the main technologies for big data networking framework is online multihoming optimization that is large-scale dimension table association technology in a distributed environment. It is often used in applications like real-time suggestion and research. Big data is concerned with the quality of large datasets that are distributed. These datasets demand sophisticated network technologies to adequately transmit massive share files. Dimension table association is the process of integrating multihoming stream data with offline stored dimension table data and executing data processing using novel big data frameworks, as described in this study. The current technological options for dimension table connection are assessed first, followed by accompanying optimization technologies and the design route of mainstream distributed engines. The dimension table data query is the one that has been optimized with the greatest performance. Nonetheless, the typical optimization approach is influenced by the dimension, table size, and the design route of the mainstream distributed engine—limits on data flow rate. Second, due to the limitations of existing optimization technologies for the overall consideration of the cluster in a distributed environment, a computing model suited for hybrid computing of offline batch data and real-time streaming data is provided, followed by a single-point reading. Dimension table data, the dimension table associated data technique for distribution and calculation after segmentation, and optimization of the dimension table associated calculation logic adapt to a larger dimension table scale and are no longer restricted to data connections. Since optimizing the query of dimension table, data is employed to reduce the I/O overhead and delay caused by querying dimension table in big data. Finally, both the suggested and standard dimension table association technologies are implemented on the Apache Flink stream computing engine. Through trials, the throughput and latency on data created by Alibaba’s “Double Eleven” are compared, demonstrating the usefulness of dimension table association techniques for Distributed Stream Computing optimization by utilizing multihoming networks.