Stochastic Approximation-Based Transport Profiling for Big Data Movement Over Dedicated Connections

Abstract

High-performance networks featuring advance bandwidth reservation have been developed and deployed to support big data transfer in extreme-scale scientific applications. The performance of such big data transfer largely depends on the transport protocols being used. For a given protocol in a given network environment, different parameter settings may lead to different performance, and oftentimes the default settings do not yield the best performance. It is, however, impractical to conduct an exhaustive search in the large parameter space of transport protocols for a set of suitable parameter values. This chapter proposes a stochastic approximation-based transport profiler, namely FastProf, to quickly determine the optimal operational zone of a protocol over dedicated connections. The proposed method is evaluated using both emulations based on real-life measurements and experiments over physical connections. The results show that FastProf significantly reduces the profiling overhead while achieving a comparable level of transport performance with the exhaustive search-based approach.