Computing Quantiles of Functions of the Agent Distribution Using t-Digests

Abstract

AbstractWe introduce t-Digests as an accurate and computationally efficient way to calculate the quantiles of functions of the agent distribution for models in which the full distribution is too large to work with directly; e.g., to calculate the Top 1% of the wealth distribution. When it is possible to fit the entire agent distribution (and a function evaluation on it) into memory, the quantiles can easily be calculated directly. Evaluating a function on the agent distribution can be done much faster using a GPU, however this frequently introduces a memory bottleneck as GPU memory is typically an order of magnitude smaller than CPU memory. For Heterogeneous Agent models the full distribution of agents may fit in CPU memory, but not in GPU memory. We partition the agent distribution into (non-overlapping) subspaces and because these subspaces are much smaller we can fit the subspaces in GPU memory one at a time. For each subspace we then evaluate the function and calculate a t-Digest. t-Digests are a form of data structure that are fast to compute, require less memory, and provide high accuracy for the quantiles of the distribution; t-Digests are a ’sketch’ of the subspace. Having computed and stored a t-Digest for each subspace we can then merge them to get a t-Digest for the full agent distribution. The resulting t-Digest is an accurate representation of the quantiles and can form the basis of Lorenz Curves as well as many other distributional and inequality statistics of interest in Economics.