An optimization approach to supervised principal component analysis

Abstract

Supervised dimensionality reduction has become an important theme in the last two decades. Despite the plethora of models and formulations, there is a lack of a simple model that aims to project the set of patterns into a space defined by the classes (or categories). We set up a model where each class is represented as a 1D subspace of the vector space formed by the features. Assuming the set of classes does not exceed the cardinality of the features, the model results in multi-class supervised learning in which the features of each class are projected into the class subspace. Class discrimination is guaranteed via the imposition of the orthogonality of the 1D class sub-spaces. The resulting optimization problem—formulated as the minimization of a sum of quadratic functions on a Stiefel manifold—while being non-convex (due to the constraints), has a structure for which we can identify when we have reached a global minimum. After formulating a version with standard inner products, we extend the formulation to a reproducing kernel Hilbert space and similarly to the kernel version. Comparisons with the multi-class Fisher discriminants and principal component analysis showcase the relative merits toward dimensionality reduction.