Network traffic characteristics of hyperscale data centers in the era of cloud applications

Abstract

We present the network architecture of Alibaba Cloud DCs and investigate their traffic characteristics based on statistical data and captured traces. The statistical coarse-grained data are in the granularity of one minute, while the captured traces are fine-grained data that are in the granularity of one packet. We study the traffic features from the perspective of a macroscopic view, network performance, and microscopic view. The results report that the average utilization ratio of spine switches is stable when the observation time period reaches one day and the intra-ToR traffic ratio is in the range of 2%–10%. By mapping the folded-Clos topology to a tree topology and considering logical switching planes, we obtain the traffic matrix among pods from the average port utilization ratio. As we further investigate the perspective of network performance and the microscopic view, we find that there is no cell loss happening as the normalized queue speedQsis lower than 0.4. The normalized queue speedQsis defined as the total bytes of a queue sent in 1 s divided by 100 Gb, which reflects the packet sending speed of the queue. The observed maximum buffer size for one port conforms with the calculated maximum buffer occupation of 2.8 MB. By analyzing the captured traces, we find that the packet length is subject to a trimodal distribution. Under a time granularity of 10 ms, the instant bandwidth of one ToR port could reach 96 Gb/s at an average load of around 0.2 under a maximum link bandwidth of 100 Gb/s.