Refinements of Universal Approximation Results for Deep Belief Networks and Restricted Boltzmann Machines

Abstract

We improve recently published results about resources of restricted Boltzmann machines (RBM) and deep belief networks (DBN) required to make them universal approximators. We show that any distribution [Formula: see text] on the set [Formula: see text] of binary vectors of length [Formula: see text] can be arbitrarily well approximated by an RBM with [Formula: see text] hidden units, where [Formula: see text] is the minimal number of pairs of binary vectors differing in only one entry such that their union contains the support set of [Formula: see text]. In important cases this number is half the cardinality of the support set of [Formula: see text] (given in Le Roux & Bengio, 2008 ). We construct a DBN with [Formula: see text], hidden layers of width [Formula: see text] that is capable of approximating any distribution on [Formula: see text] arbitrarily well. This confirms a conjecture presented in Le Roux and Bengio ( 2010 ).