Title: Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models Show Chinese title

Title: 变分专家混合自编码器用于多模态深度生成模型

Abstract: Learning generative models that span multiple data modalities, such as vision and language, is often motivated by the desire to learn more useful, generalisable representations that faithfully capture common underlying factors between the modalities. In this work, we characterise successful learning of such models as the fulfillment of four criteria: i) implicit latent decomposition into shared and private subspaces, ii) coherent joint generation over all modalities, iii) coherent cross-generation across individual modalities, and iv) improved model learning for individual modalities through multi-modal integration. Here, we propose a mixture-of-experts multimodal variational autoencoder (MMVAE) to learn generative models on different sets of modalities, including a challenging image-language dataset, and demonstrate its ability to satisfy all four criteria, both qualitatively and quantitatively. Show Chinese abstract

Abstract: 学习能够涵盖多种数据模态（如视觉和语言）的生成模型，通常是为了学习更实用、可泛化的表示，这些表示能够准确捕捉模态之间的共同底层因素。 在本工作中，我们将此类模型的成功学习定义为满足四个标准：i) 隐含潜在分解为共享和私有子空间，ii) 所有模态上的连贯联合生成，iii) 单个模态之间的连贯跨模态生成，iv) 通过多模态整合提高单个模态的模型学习。 在此，我们提出一种专家混合多模态变分自编码器（MMVAE），以在不同模态集上学习生成模型，包括一个具有挑战性的图像-语言数据集，并展示其能够同时在定性和定量方面满足所有四个标准。