标题： StreamDiT：实时流式文本到视频生成 显示英文标题

标题： StreamDiT: Real-Time Streaming Text-to-Video Generation

摘要： 最近，通过将基于Transformer的扩散模型扩展到数十亿参数，文本到视频（T2V）生成取得了重大进展，可以生成高质量的视频。 然而，现有模型通常只能离线生成短片段，限制了其在交互式和实时应用中的使用。 本文通过提出StreamDiT，一种流式视频生成模型，解决了这些挑战。 StreamDiT的训练基于流匹配，通过添加一个移动缓冲区进行训练。 我们设计了不同缓冲帧分块方案的混合训练，以提高内容一致性和视觉质量。 StreamDiT建模基于具有可变时间嵌入和窗口注意力的adaLN DiT。 为了实践所提出的方法，我们训练了一个具有40亿参数的StreamDiT模型。 此外，我们提出了一种针对StreamDiT的多步骤蒸馏方法。 在选定的分块方案的每个段中进行采样蒸馏。 蒸馏后，总函数评估次数（NFEs）减少到缓冲区中块的数量。 最后，我们的蒸馏模型在单个GPU上达到16 FPS的实时性能，可以以512p分辨率生成视频流。 我们通过定量指标和人类评估来评估我们的方法。 我们的模型实现了实时应用，例如 流式生成、交互式生成和视频到视频。 我们在项目网站上提供了视频结果和更多示例：https://cumulo-autumn.github.io/StreamDiT/ 显示英文摘要

摘要： Recently, great progress has been achieved in text-to-video (T2V) generation by scaling transformer-based diffusion models to billions of parameters, which can generate high-quality videos. However, existing models typically produce only short clips offline, restricting their use cases in interactive and real-time applications. This paper addresses these challenges by proposing StreamDiT, a streaming video generation model. StreamDiT training is based on flow matching by adding a moving buffer. We design mixed training with different partitioning schemes of buffered frames to boost both content consistency and visual quality. StreamDiT modeling is based on adaLN DiT with varying time embedding and window attention. To practice the proposed method, we train a StreamDiT model with 4B parameters. In addition, we propose a multistep distillation method tailored for StreamDiT. Sampling distillation is performed in each segment of a chosen partitioning scheme. After distillation, the total number of function evaluations (NFEs) is reduced to the number of chunks in a buffer. Finally, our distilled model reaches real-time performance at 16 FPS on one GPU, which can generate video streams at 512p resolution. We evaluate our method through both quantitative metrics and human evaluation. Our model enables real-time applications, e.g. streaming generation, interactive generation, and video-to-video. We provide video results and more examples in our project website: https://cumulo-autumn.github.io/StreamDiT/