Faithful Representation of Separable Distributions

Abstract

A geometric approach to data representation incorporating information theoretic ideas is presented. The task of finding a faithful representation, where the input distribution is evenly partitioned into regions of equal mass, is addressed. For input consisting of mixtures of statistically independent sources, we treat independent component analysis (ICA) as a computational geometry problem. First, we consider the separation of sources with sharply peaked distribution functions, where the ICA problem becomes that of finding high-density directions in the input distribution. Second, we consider the more general problem for arbitrary input distributions, where ICA is transformed into the task of finding an aligned equipartition. By modifying the Kohonen self-organized feature maps, we arrive at neural networks with local interactions that optimize coding while simultaneously performing source separation. The local nature of our approach results in networks with nonlinear ICA capabilities.