标题： 稀疏-vDiT：释放稀疏注意力的力量以加速视频扩散变换器 显示英文标题

标题： Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformers

摘要： 虽然扩散Transformer（DiTs）在视频生成方面取得了突破，但这一长序列生成任务仍然受到注意力机制二次复杂性的限制，导致显著的推理延迟。 通过对视频扩散Transformer（vDiT）中注意力图的详细分析，我们确定了三种重复出现的稀疏模式：对角线、多对角线和竖条纹结构。 并且甚至可以跳过3-6%的注意力头。 至关重要的是，这些模式表现出强烈的层深度和头位置相关性，但对输入内容的依赖有限。 利用这些发现，我们提出了Sparse-vDiT，这是一种针对vDiT的稀疏加速框架，包括：1）针对每种识别出的稀疏模式，用计算效率更高的实现替换密集注意力的模式优化稀疏核。2）一种离线稀疏扩散搜索算法，通过硬件感知成本建模选择每层和每个头的最佳稀疏计算策略。 在确定最佳配置后，我们将同一层内共享相同注意力策略的头融合在一起，从而提高推理效率。 集成到最先进的vDiT模型（CogVideoX1.5、HunyuanVideo和Wan2.1）中，Sparse-vDiT分别实现了2.09$\times$、2.38$\times$和 1.67$\times$的理论浮点运算（FLOP）减少，以及实际推理加速比1.76$\times$、1.85$\times$和 1.58$\times$，同时保持高视觉保真度，PSNR值达到24.13、27.09 和 22.59。 我们的工作表明，vDiTs中的潜在结构稀疏性可以系统地用于长视频合成。 显示英文摘要

摘要： While Diffusion Transformers (DiTs) have achieved breakthroughs in video generation, this long sequence generation task remains constrained by the quadratic complexity of attention mechanisms, resulting in significant inference latency. Through detailed analysis of attention maps in Video Diffusion Transformer (vDiT), we identify three recurring sparsity patterns: diagonal, multi-diagonal, and vertical-stripe structures. And even 3-6\% attention heads can be skipped. Crucially, these patterns exhibit strong layer-depth and head-position correlations but show limited dependence on the input content. Leveraging these findings, we propose Sparse-vDiT, a sparsity acceleration framework for vDiT comprising: 1) Pattern-optimized sparse kernels that replace dense attention with computationally efficient implementations for each identified sparsity pattern. 2) An offline sparse diffusion search algorithm that selects the optimal sparse computation strategy per layer and head via hardware-aware cost modeling. After determining the optimal configuration, we fuse heads within the same layer that share the same attention strategy, enhancing inference efficiency. Integrated into state-of-the-art vDiT models (CogVideoX1.5, HunyuanVideo, and Wan2.1), Sparse-vDiT achieves 2.09$\times$, 2.38$\times$, and 1.67$\times$ theoretical FLOP reduction, and actual inference speedups of 1.76$\times$, 1.85$\times$, and 1.58$\times$, respectively, while maintaining high visual fidelity, with PSNR values reaching 24.13, 27.09, and 22.59. Our work demonstrates that latent structural sparsity in vDiTs can be systematically exploited for long video synthesis.