Affiliation:
1. Complutense University of Madrid
2. University of Paris-Saclay
Abstract
Restricted Boltzmann Machines are simple and powerful generative
models that can encode any complex dataset. Despite all their
advantages, in practice the trainings are often unstable and it is
difficult to assess their quality because the dynamics are affected by
extremely slow time dependencies. This situation becomes critical when
dealing with low-dimensional clustered datasets, where the time required
to sample ergodically the trained models becomes computationally
prohibitive. In this work, we show that this divergence of Monte Carlo
mixing times is related to a phenomenon of phase coexistence, similar to
that which occurs in physics near a first-order phase transition. We
show that sampling the equilibrium distribution using the Markov chain
Monte Carlo method can be dramatically accelerated when using biased
sampling techniques, in particular the Tethered Monte Carlo (TMC)
method. This sampling technique efficiently solves the problem of
evaluating the quality of a given trained model and generating new
samples in a reasonable amount of time. Moreover, we show that this
sampling technique can also be used to improve the computation of the
log-likelihood gradient during training, leading to dramatic
improvements in training RBMs with artificial clustered datasets. On
real low-dimensional datasets, this new training method fits RBM models
with significantly faster relaxation dynamics than those obtained with
standard PCD recipes. We also show that TMC sampling can be used to
recover the free-energy profile of the RBM. This proves to be extremely
useful to compute the probability distribution of a given model and to
improve the generation of new decorrelated samples in slow PCD-trained
models. The main limitations of this method are, first, the restriction
to effective low-dimensional datasets and, second, the fact that the
Tethered MC method breaks the possibility of performing parallel
alternative Monte Carlo updates, which limits the size of the systems we
can consider in practice.
Funder
Banco Santander
Comunidad de Madrid
Ministerio de Economía y Competitividad
Universidad Complutense de Madrid
Subject
General Physics and Astronomy
Cited by
3 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献