Abstract
We prove a quantitative functional central limit theorem for one-hidden-layer neural networks with generic activation function. Our rates of convergence depend heavily on the smoothness of the activation function, and they range from logarithmic for nondifferentiable nonlinearities such as the ReLu to $\sqrt{n}$ for highly regular activations. Our main tools are based on functional versions of the Stein–Malliavin method; in particular, we rely on a quantitative functional central limit theorem which has been recently established by Bourguin and Campese [Electron. J. Probab. 25 (2020), 150].
Subject
Statistics, Probability and Uncertainty,Modeling and Simulation,Statistics and Probability
Reference30 articles.
1. Malliavin-Stein method: a survey of some recent developments;Mod. Stoch. Theory Appl.,2021
2. Breaking the curse of dimensionality with convex neural networks;J. Mach. Learn. Res.,2017
3. Reconciling modern machine-learning practice and the classical bias-variance trade-off;Proc. Natl. Acad. Sci. USA,2019