标题： SkyReels-音频：视频扩散变换器中的全方位音频条件化说话人肖像 显示英文标题

标题： SkyReels-Audio: Omni Audio-Conditioned Talking Portraits in Video Diffusion Transformers

摘要： 受文本、图像和视频等多种模态输入引导的音频驱动的可操控肖像生成与编辑仍是一个未被充分探索的领域。本文提出了一种名为SkyReels-Audio的统一框架，用于合成高保真且时间上一致的可操控肖像视频。该框架基于预训练的视频扩散变换器构建，支持无限长度的生成与编辑，同时通过多模态输入实现多样化且可控的条件设置。我们采用混合课程学习策略逐步对齐音频与面部运动，从而实现对长视频序列的细粒度多模态控制。为了增强局部面部一致性，我们引入了面部掩码损失和音频引导的无分类器引导机制。滑动窗口去噪方法进一步融合了时间片段中的潜在表示，确保在扩展时间和多样身份下具有视觉保真度和时间一致性。更重要的是，我们构建了一个专用的数据管道，用于整理同步的音频、视频和文本描述组成的高质量三元组。综合基准评估表明，SkyReels-Audio在唇音同步准确性、身份一致性以及逼真的面部动态方面表现出色，尤其是在复杂和具有挑战性的条件下。 显示英文摘要

摘要： The generation and editing of audio-conditioned talking portraits guided by multimodal inputs, including text, images, and videos, remains under explored. In this paper, we present SkyReels-Audio, a unified framework for synthesizing high-fidelity and temporally coherent talking portrait videos. Built upon pretrained video diffusion transformers, our framework supports infinite-length generation and editing, while enabling diverse and controllable conditioning through multimodal inputs. We employ a hybrid curriculum learning strategy to progressively align audio with facial motion, enabling fine-grained multimodal control over long video sequences. To enhance local facial coherence, we introduce a facial mask loss and an audio-guided classifier-free guidance mechanism. A sliding-window denoising approach further fuses latent representations across temporal segments, ensuring visual fidelity and temporal consistency across extended durations and diverse identities. More importantly, we construct a dedicated data pipeline for curating high-quality triplets consisting of synchronized audio, video, and textual descriptions. Comprehensive benchmark evaluations show that SkyReels-Audio achieves superior performance in lip-sync accuracy, identity consistency, and realistic facial dynamics, particularly under complex and challenging conditions.