Iterative Retrieval and Block Coding in Autoassociative and Heteroassociative Memory

Abstract

Neural associative memories (NAM) are perceptron-like single-layer networks with fast synaptic learning typically storing discrete associations between pairs of neural activity patterns. Gripon and Berrou ( 2011 ) investigated NAM employing block coding, a particular sparse coding method, and reported a significant increase in storage capacity. Here we verify and extend their results for both heteroassociative and recurrent autoassociative networks. For this we provide a new analysis of iterative retrieval in finite autoassociative and heteroassociative networks that allows estimating storage capacity for random and block patterns. Furthermore, we have implemented various retrieval algorithms for block coding and compared them in simulations to our theoretical results and previous simulation data. In good agreement of theory and experiments, we find that finite networks employing block coding can store significantly more memory patterns. However, due to the reduced information per block pattern, it is not possible to significantly increase stored information per synapse. Asymptotically, the information retrieval capacity converges to the known limits [Formula: see text] and [Formula: see text] also for block coding. We have also implemented very large recurrent networks up to [Formula: see text] neurons, showing that maximal capacity [Formula: see text] bit per synapse occurs for finite networks having a size [Formula: see text] similar to cortical macrocolumns.