The effective number of shared dimensions: A simple method for revealing shared structure between datasets

Abstract

AbstractA number of recent studies have sought to understand the behavior of artificial and biological neural networks by comparing representations across layers, networks and brain areas. Simultaneously, there has been growing interest in using dimensionality of a dataset as a proxy for computational complexity. At the intersection of these topics, studies exploring the dimensionality of shared computational and representational subspaces have relied on model-based methods, but a standard, model-free measure is lacking. Here we present a candidate measure for shared dimensionality that we call the effective number of shared dimensions (ENSD). The ENSD can be applied to data matrices sharing at least one dimension, reduces to the well-known participation ratio when both data sets are equivalent and has a number of other robust and intuitive mathematical properties. Notably, the ENSD can be written as a similarity metric that is a re-scaled version of centered kernel alignment (CKA) but additionally describes the dimensionality of the aligned subspaces. Unlike methods like canonical correlation analysis (CCA), the ENSD is robust to cases where data is sparse or low rank. We demonstrate its utility and computational efficiency by a direct comparison of CKA and ENSD on across-layer similarities in convolutional neural networks as well as by recovering results from recent studies in neuroscience on communication subspaces between brain regions. Finally, we demonstrate how the ENSD and its constituent statistics allow us to perform a variety of multi-modal analyses of multivariate datasets. Specifically, we use connectomic data to probe the alignment of parallel pathways in the fly olfactory system, revealing novel results in the interaction between innate and learned olfactory representations. Altogether, we show that the ENSD is an interpretable and computationally efficient model-free measure of shared dimensionality and that it can be used to probe shared structure in a wide variety of data types.