Investigating Unsupervised Neural Machine Translation for Low-resource Language Pair English-Mizo via Lexically Enhanced Pre-trained Language Models

Abstract

The vast majority of languages in the world at present are considered to be low-resource languages. Since the availability of large parallel data is crucial for the success of most modern machine translation approaches, improving machine translation for low-resource languages is a key challenge. Most unsupervised techniques for translation benefit closely related languages with monolingual data of substantial quantity. To facilitate research in this direction for the extremely low resource language pair English ( en ) and Mizo ( lus ), we have developed a parallel and monolingual corpus for the Mizo language from various news websites. We explore Unsupervised Neural Machine Translation (UNMT) based on the developed monolingual data. We observe that cross-lingual embedding (CLWE) initializations on subword segmented data during pre-training, based on both masked language modelling and sequence-to-sequence generation tasks, improve translation performance. We experiment with cross-lingual alignment and combined alignment and joint training for learning the cross-lingual embedding representations. We also report baseline performances and the impact of CLWE initialization using semi-supervised and supervised neural machine translation. Empirical results show that both CLWE initializations work well for the distant pair English-Mizo compared to the baselines.