Title: LTX-Video: Realtime Video Latent Diffusion Show Chinese title

Title: LTX-视频：实时视频潜在扩散

Abstract: We introduce LTX-Video, a transformer-based latent diffusion model that adopts a holistic approach to video generation by seamlessly integrating the responsibilities of the Video-VAE and the denoising transformer. Unlike existing methods, which treat these components as independent, LTX-Video aims to optimize their interaction for improved efficiency and quality. At its core is a carefully designed Video-VAE that achieves a high compression ratio of 1:192, with spatiotemporal downscaling of 32 x 32 x 8 pixels per token, enabled by relocating the patchifying operation from the transformer's input to the VAE's input. Operating in this highly compressed latent space enables the transformer to efficiently perform full spatiotemporal self-attention, which is essential for generating high-resolution videos with temporal consistency. However, the high compression inherently limits the representation of fine details. To address this, our VAE decoder is tasked with both latent-to-pixel conversion and the final denoising step, producing the clean result directly in pixel space. This approach preserves the ability to generate fine details without incurring the runtime cost of a separate upsampling module. Our model supports diverse use cases, including text-to-video and image-to-video generation, with both capabilities trained simultaneously. It achieves faster-than-real-time generation, producing 5 seconds of 24 fps video at 768x512 resolution in just 2 seconds on an Nvidia H100 GPU, outperforming all existing models of similar scale. The source code and pre-trained models are publicly available, setting a new benchmark for accessible and scalable video generation. Show Chinese abstract

Abstract: 我们介绍了LTX-Video，这是一种基于变换器的潜在扩散模型，通过无缝整合Video-VAE和去噪变换器的功能，以整体方法生成视频。与现有方法不同，这些方法将这些组件视为独立的，LTX-Video旨在优化它们的交互以提高效率和质量。其核心是一个精心设计的Video-VAE，实现了1:192的高压缩比，每个标记的空间时间下采样为32×32×8像素，这是通过将补丁操作从变换器的输入重新定位到VAE的输入实现的。在这个高度压缩的潜在空间中运行使变换器能够高效地执行完整的空间时间自注意力，这对于生成具有时间一致性的高分辨率视频至关重要。然而，高压缩率本身限制了精细细节的表示。为了解决这个问题，我们的VAE解码器负责潜伏到像素转换以及最终的去噪步骤，在像素空间中直接产生清晰的结果。这种方法保留了生成精细细节的能力，而不会带来单独上采样模块的运行时成本。我们的模型支持多种用例，包括文本到视频和图像到视频生成，这两种能力同时训练。它实现了比实时更快的生成速度，在Nvidia H100 GPU上以768×512分辨率生成24帧/秒的5秒视频仅需2秒，优于所有类似规模的现有模型。源代码和预训练模型公开可用，为可访问和可扩展的视频生成设定了新的基准。