An Improved Semi-supervised Variational Autoencoder with Gate Mechanism for Text Classification

Abstract

In recent years, semi-supervised learning has been investigated to take full advantages of increasing unlabeled data. Although pretrained deep learning models are successfully adopted on a massive amount of unlabeled data, they may not be applicable in specific domains as the data is limited. In this paper, we propose a model, termed Semi-supervised Variational AutoEncoder (SVAE), which consists of Gated Convolutional Neural Networks (GCNN) as both the encoder and the decoder. Since the canonical VAE suffers from Kullback–Leibler (KL) vanishing problem, we attach a layer named Scalar after Batch Normalization (BN) to scale the output of the BN. We conduct experiments on two domain-specific datasets with a small amount of data. The results show that SVAE outperforms other alternative baselines for language modeling and semi-supervised learning studies. Especially, the results in the language modeling validate the effect of combining BN and Scalar for tackling the KL vanishing problem. Moreover, the visualization of the latent representations verifies the performance of SVAE on less data.