标题： MoDiT：使用扩散变换器学习高度一致的3D运动系数以生成说话头像 显示英文标题

标题： MoDiT: Learning Highly Consistent 3D Motion Coefficients with Diffusion Transformer for Talking Head Generation

摘要： 音频驱动的说话头部生成对于虚拟助手、视频游戏和电影等应用至关重要，其中自然的嘴唇运动是关键。 尽管在这一领域取得了进展，但在生成一致且逼真的面部动画方面仍存在挑战。 现有方法通常基于生成对抗网络（GANs）或基于UNet的扩散模型，面临三个主要限制：(i) 由于时间约束较弱导致的时间抖动，造成帧不一致；(ii) 由于缺乏足够的三维信息提取导致的身份漂移，进而导致面部身份保存不佳；以及(iii) 由于对真实眨眼动态建模不足导致的不自然眨眼行为。 为了解决这些问题，我们提出了MoDiT，一种将三维可变形模型（3DMM）与基于扩散的Transformer相结合的新框架。 我们的贡献包括：(i) 一种分层去噪策略，结合改进的时间注意力和有偏的自注意力/交叉注意力机制，使模型能够优化嘴唇同步并逐步增强全脸一致性，有效缓解时间抖动。 (ii) 集成3DMM系数以提供显式的空间约束，确保准确的三维引导光流预测，并利用Wav2Lip结果改善嘴唇同步，从而保持身份一致性。 (iii) 一种改进的眨眼策略，用于建模自然的眼部运动，具有更平滑和逼真的眨眼行为。 显示英文摘要

摘要： Audio-driven talking head generation is critical for applications such as virtual assistants, video games, and films, where natural lip movements are essential. Despite progress in this field, challenges remain in producing both consistent and realistic facial animations. Existing methods, often based on GANs or UNet-based diffusion models, face three major limitations: (i) temporal jittering caused by weak temporal constraints, resulting in frame inconsistencies; (ii) identity drift due to insufficient 3D information extraction, leading to poor preservation of facial identity; and (iii) unnatural blinking behavior due to inadequate modeling of realistic blink dynamics. To address these issues, we propose MoDiT, a novel framework that combines the 3D Morphable Model (3DMM) with a Diffusion-based Transformer. Our contributions include: (i) A hierarchical denoising strategy with revised temporal attention and biased self/cross-attention mechanisms, enabling the model to refine lip synchronization and progressively enhance full-face coherence, effectively mitigating temporal jittering. (ii) The integration of 3DMM coefficients to provide explicit spatial constraints, ensuring accurate 3D-informed optical flow prediction and improved lip synchronization using Wav2Lip results, thereby preserving identity consistency. (iii) A refined blinking strategy to model natural eye movements, with smoother and more realistic blinking behaviors.