标题： 通过下一帧扩散在30+ FPS下玩转Transformer 显示英文标题

标题： Playing with Transformer at 30+ FPS via Next-Frame Diffusion

摘要： 自回归视频模型在创建交互式视频内容和支持任意持续时间的流媒体应用方面相较于双向扩散模型具有明显的优势。 在本工作中，我们提出了下一帧扩散（NFD），这是一种自回归扩散变压器，结合了逐块因果注意力，通过每帧内的并行标记生成实现迭代采样和高效推理。 然而，实现实时视频生成对于此类模型仍然是一个重大挑战，主要是由于扩散采样的高计算成本以及自回归生成固有的硬件效率低下。 为了解决这个问题，我们引入了两项创新：（1）我们将一致性蒸馏扩展到视频领域，并专门为其调整以适用于视频模型，从而实现使用少量采样步骤的高效推理；（2）为了充分利用并行计算，受观察到相邻帧通常共享相同的动作输入的启发，我们提出了推测采样。 在这种方法中，模型使用当前的动作输入生成接下来的几帧，并在输入动作不同时丢弃推测生成的帧。 在大规模动作条件视频生成基准上的实验表明，NFD在视觉质量和采样效率方面均优于自回归基线。 我们首次在A100 GPU上使用310M模型实现了每秒超过30帧（FPS）的自回归视频生成。 显示英文摘要

摘要： Autoregressive video models offer distinct advantages over bidirectional diffusion models in creating interactive video content and supporting streaming applications with arbitrary duration. In this work, we present Next-Frame Diffusion (NFD), an autoregressive diffusion transformer that incorporates block-wise causal attention, enabling iterative sampling and efficient inference via parallel token generation within each frame. Nonetheless, achieving real-time video generation remains a significant challenge for such models, primarily due to the high computational cost associated with diffusion sampling and the hardware inefficiencies inherent to autoregressive generation. To address this, we introduce two innovations: (1) We extend consistency distillation to the video domain and adapt it specifically for video models, enabling efficient inference with few sampling steps; (2) To fully leverage parallel computation, motivated by the observation that adjacent frames often share the identical action input, we propose speculative sampling. In this approach, the model generates next few frames using current action input, and discard speculatively generated frames if the input action differs. Experiments on a large-scale action-conditioned video generation benchmark demonstrate that NFD beats autoregressive baselines in terms of both visual quality and sampling efficiency. We, for the first time, achieves autoregressive video generation at over 30 Frames Per Second (FPS) on an A100 GPU using a 310M model.