Multi-Modal Latent Diffusion
Author:
Bounoua Mustapha12ORCID, Franzese Giulio2ORCID, Michiardi Pietro2ORCID
Affiliation:
1. Ampere Software Technology, 06560 Valbonne, France 2. Department of Data Science, EURECOM, 06410 Biot, France
Abstract
Multimodal datasets are ubiquitous in modern applications, and multimodal Variational Autoencoders are a popular family of models that aim to learn a joint representation of different modalities. However, existing approaches suffer from a coherence–quality tradeoff in which models with good generation quality lack generative coherence across modalities and vice versa. In this paper, we discuss the limitations underlying the unsatisfactory performance of existing methods in order to motivate the need for a different approach. We propose a novel method that uses a set of independently trained and unimodal deterministic autoencoders. Individual latent variables are concatenated into a common latent space, which is then fed to a masked diffusion model to enable generative modeling. We introduce a new multi-time training method to learn the conditional score network for multimodal diffusion. Our methodology substantially outperforms competitors in both generation quality and coherence, as shown through an extensive experimental campaign.
Funder
European Commission
Reference66 articles.
1. He, R., Sun, S., Yu, X., Xue, C., Zhang, W., Torr, P., Bai, S., and QI, X. Is Synthetic Data from Generative Models Ready for Image Recognition? In Proceedings of the Eleventh International Conference on Learning Representations, Kigali, Rwanda, 1–5 May 2023. 2. Azizi, S., Kornblith, S., Saharia, C., Norouzi, M., and Fleet, D.J. (2023). Synthetic Data from Diffusion Models Improves ImageNet Classification. arXiv. 3. Sariyildiz, M.B., Alahari, K., Larlus, D., and Kalantidis, Y. (2023). Fake it till you make it: Learning transferable representations from synthetic ImageNet clones. arXiv. 4. Antelmi, L., Ayache, N., Robert, P., and Lorenzi, M. (2019, January 9–15). Sparse Multi-Channel Variational Autoencoder for the Joint Analysis of Heterogeneous Data. Proceedings of the 36th International Conference on Machine Learning, Long Beach, CA, USA. 5. Da Silva–Filarder, M., Ancora, A., Filippone, M., and Michiardi, P. (2021, January 13–16). Multimodal Variational Autoencoders for Sensor Fusion and Cross Generation. Proceedings of the 2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA), Virtual.
|
|