标题： MoDA：用于说话头生成的多模态扩散架构 显示英文标题

标题： MoDA: Multi-modal Diffusion Architecture for Talking Head Generation

摘要： 以任意身份和语音音频生成对话头像仍然是虚拟元宇宙领域的一个关键问题。 最近，扩散模型因其强大的生成能力而成为该领域的流行生成技术。 然而，基于扩散的方法仍面临几个挑战：1）由变分自编码器（VAE）的隐式潜在空间引起的推理效率低下和视觉伪影，这使得扩散过程复杂化；2）由于多模态信息融合不足，导致缺乏真实的面部表情和头部动作。 在本文中，MoDA通过以下方式解决这些挑战：1）定义一个联合参数空间，连接运动生成和神经渲染，并利用流匹配来简化扩散学习；2）引入一种多模态扩散架构，对噪声运动、音频和辅助条件之间的相互作用进行建模，从而增强整体面部表现力。 此外，采用了一种从粗到细的融合策略，逐步整合不同模态，确保有效的特征融合。 实验结果表明，MoDA提高了视频的多样性、真实性和效率，使其适用于实际应用。 项目页面：https://lixinyyang.github.io/MoDA.github.io/ 显示英文摘要

摘要： Talking head generation with arbitrary identities and speech audio remains a crucial problem in the realm of the virtual metaverse. Recently, diffusion models have become a popular generative technique in this field with their strong generation capabilities. However, several challenges remain for diffusion-based methods: 1) inefficient inference and visual artifacts caused by the implicit latent space of Variational Auto-Encoders (VAE), which complicates the diffusion process; 2) a lack of authentic facial expressions and head movements due to inadequate multi-modal information fusion. In this paper, MoDA handles these challenges by: 1) defining a joint parameter space that bridges motion generation and neural rendering, and leveraging flow matching to simplify diffusion learning; 2) introducing a multi-modal diffusion architecture to model the interaction among noisy motion, audio, and auxiliary conditions, enhancing overall facial expressiveness. In addition, a coarse-to-fine fusion strategy is employed to progressively integrate different modalities, ensuring effective feature fusion. Experimental results demonstrate that MoDA improves video diversity, realism, and efficiency, making it suitable for real-world applications. Project Page: https://lixinyyang.github.io/MoDA.github.io/