An Orthonormalization-Free and Inversion-Free Algorithm for Online Estimation of Principal Eigenspace

Abstract

In this paper, we study the problem of estimating the principal eigenspace over an online data stream. First-order optimization methods are appealing choices for this problem thanks to their high efficiency and easy implementation. The existing first-order solvers, however, require either per-step orthonormalization or matrix inversion, which empirically puts pressure to the parameter tuning, and also incurs extra costs of rank augmentation. To get around these limitations, we introduce a penalty-like term controlling the distance from the Stiefel manifold into matrix Krasulina’s method, and propose the first orthonormalization- and inversion-free incremental PCA scheme (Domino). The Domino is shown to achieve the computational speed-up, and own the ability of automatic correction on the numerical rank. It also maintains the advantage of Krasulina’s method, e.g., variance reduction on low-rank data. Moreover, both of the asymptotic and non-asymptotic convergence guarantees are established for the proposed algorithm.