THE SPECTRUM OF RANDOM INNER-PRODUCT KERNEL MATRICES

Abstract

We consider n × n matrices whose (i, j)th entry is [Formula: see text], where X1, …, Xn are i.i.d. standard Gaussian in ℝp, and f is a real-valued function. The weak limit of the eigenvalue distribution of these random matrices is studied at the limit when p, n → ∞ and p/n = γ which is a constant. We show that, under certain conditions on the kernel function f, the limiting spectral density exists and is dictated by a cubic equation involving its Stieltjes transform. The parameters of this cubic equation are decided by a Hermite-like expansion of the rescaled kernel function. While the case that f is differentiable at the origin has been previously resolved by El Karoui [The spectrum of kernel random matrices, Ann. Statist.38 (2010) 1–50], our result is applicable to non-smooth f, such as the Sign function and the hard thresholding operator of sample covariance matrices. For this larger class of kernel functions, we obtain a new family of limiting densities, which includes the Marčenko–Pastur (M.P.) distribution and Wigner's semi-circle distribution as special cases.