A Co-Plot analysis of logs and models of parallel workloads

Abstract

We present a multivariate analysis technique called Co-Plot that is especially suitable for few samples of many variables. Co-Plot embeds the multidimensional samples in two dimensions, in a way that allows key variables to be identified, and relations between both variables and observations to be analyzed together. When applied to the workloads on parallel supercomputers, we find two stable perpendicular axes of highly correlated variables, one representing individual job attributes and the other representing multijob attributes. The different workloads, on the other hand, are rather different from one another, and may also change over time. Synthetic models for workload generation are also analyzed, and found to be reasonable in the sense that they span the same range of variable combinations as the real workloads. However, the spread of real workloads implies that a single model cannot be similar to all of them. This leads us to construct a parameterized model, with parameters that correspond to the two axes identified above. We also find that existing models do not model the temporal structure of the workload well, and hence are wanting for tasks such as comparing schedulers, and that the common methodology for load manipulation of workloads is problematic.