High-Quality Many-to-Many Voice Conversion Using Transitive Star Generative Adversarial Networks with Adaptive Instance Normalization

Abstract

This paper proposes a novel high-quality nonparallel many-to-many voice conversion method based on transitive star generative adversarial networks with adaptive instance normalization (Trans-StarGAN-VC with AdaIN). First, we improve the structure of generator with TransNets to make full use of hierarchical features associated with speech naturalness. In TransNets, many shortcut connections share hierarchical features between encoding and decoding part to capture sufficient linguistic and semantic information, which helps to provide natural sounding converted speech and accelerate the convergence of training process. Second, by incorporating AdaIN for style transfer, we enable the generator to learn sufficient speaker characteristic information directly from speech instead of using attribute labels, which also provides a promising framework for one-shot VC. Objective and subjective experiments with nonparallel training data show that our method significantly outperforms StarGAN-VC in both speech naturalness and speaker similarity. The mean values of mean opinion score (MOS) and ABX are increased by 24.5% and 10.7%, respectively. The comparison of spectrogram also shows that our method can provide more complete harmonic structures and details, and effectively bridge the gap between converted speech and target speech.