Author:
Fares Mireille,Pelachaud Catherine,Obin Nicolas
Abstract
Modeling virtual agents with behavior style is one factor for personalizing human-agent interaction. We propose an efficient yet effective machine learning approach to synthesize gestures driven by prosodic features and text in the style of different speakers including those unseen during training. Our model performs zero-shot multimodal style transfer driven by multimodal data from the PATS database containing videos of various speakers. We view style as being pervasive; while speaking, it colors the communicative behaviors expressivity while speech content is carried by multimodal signals and text. This disentanglement scheme of content and style allows us to directly infer the style embedding even of a speaker whose data are not part of the training phase, without requiring any further training or fine-tuning. The first goal of our model is to generate the gestures of a source speaker based on thecontentof two input modalities–Mel spectrogram and text semantics. The second goal is to condition the source speaker's predicted gestures on the multimodal behaviorstyleembedding of a target speaker. The third goal is to allow zero-shot style transfer of speakers unseen during training without re-training the model. Our system consists of two main components: (1) aspeaker style encoder networkthat learns to generate a fixed-dimensional speaker embeddingstylefrom a target speaker multimodal data (mel-spectrogram, pose, and text) and (2) asequence-to-sequence synthesis networkthat synthesizes gestures based on thecontentof the input modalities—text and mel-spectrogram—of a source speaker and conditioned on the speaker style embedding. We evaluate that our model is able to synthesize gestures of a source speaker given the two input modalities and transfer the knowledge of target speaker style variability learned by the speaker style encoder to the gesture generation task in a zero-shot setup, indicating that the model has learned a high-quality speaker representation. We conduct objective and subjective evaluations to validate our approach and compare it with baselines.
Reference43 articles.
1. “Low-resource adaptation for personalized co-speech gesture generation,”;Ahuja;Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),2022
2. “Style transfer for co-speech gesture animation: A multi-speaker conditional-mixture approach,”;Ahuja,2020
3. To react or not to react: End-to-end visual pose forecasting for personalized avatar during dyadic conversations,”;Ahuja;2019 International Conference on Multimodal Interaction,2019
4. “Style-controllable speech-driven gesture synthesis using normalising flows,”;Alexanderson,2020
5. Language style as audience design;Bell;Langu. Soc,1984
Cited by
3 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献