Affiliation:
1. Shanghai Center for Mathematical Sciences , Fudan University, Shanghai 200433, China
2. Shanghai Artificial Intelligence Laboratory , Shanghai 200232, China
Abstract
Abstract
We study stochastic zeroth-order gradient and Hessian estimators for real-valued functions in $\mathbb{R}^n$. We show that, via taking finite difference along random orthogonal directions, the variance of the stochastic finite difference estimators can be significantly reduced. In particular, we design estimators for smooth functions such that, if one uses $ \varTheta \left ( k \right ) $ random directions sampled from the Stiefel manifold $ \text{St} (n,k) $ and finite-difference granularity $\delta $, the variance of the gradient estimator is bounded by $ \mathscr{O} \left ( \left ( \frac{n}{k} - 1 \right ) + \left ( \frac{n^2}{k} - n \right ) \delta ^2 + \frac{ n^2 \delta ^4} { k } \right ) $, and the variance of the Hessian estimator is bounded by $\mathscr{O} \left ( \left ( \frac{n^2}{k^2} - 1 \right ) + \left ( \frac{n^4}{k^2} - n^2 \right ) \delta ^2 + \frac{n^4 \delta ^4 }{k^2} \right ) $. When $k = n$, the variances become negligibly small. In addition, we provide improved bias bounds for the estimators. The bias of both gradient and Hessian estimators for smooth function $f$ is of order $\mathscr{O} \big( \delta ^2 \varGamma \big )$, where $\delta $ is the finite-difference granularity, and $ \varGamma $ depends on high-order derivatives of $f$. Our results are evidenced by empirical observations.
Publisher
Oxford University Press (OUP)
Subject
Applied Mathematics,Computational Theory and Mathematics,Numerical Analysis,Statistics and Probability,Analysis
Cited by
3 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献