标题： 扩散语言模型的收敛理论：信息论视角 显示英文标题

标题： A Convergence Theory for Diffusion Language Models: An Information-Theoretic Perspective

摘要： 扩散模型作为现代生成建模的强大范式已经出现，在大型语言模型（LLM）中展示了强大的潜力。与传统的自回归（AR）模型按顺序生成标记不同，扩散模型允许并行采样标记，从而实现更快的生成，并消除了从左到右生成的约束。尽管它们在实验上取得了成功，但扩散模型方法的理论理解仍然不够充分。在这项工作中，我们从信息论的角度为扩散语言模型开发了收敛保证。我们的分析表明，通过Kullback-Leibler（KL）散度测量的采样误差随着迭代次数 $T$ 的增加而呈反比衰减，并且与目标文本序列中标记之间的互信息呈线性关系。特别是，我们建立了匹配的上下界，直到某个常数因子，以证明我们收敛分析的紧密性。这些结果为扩散语言模型的实际有效性提供了新的理论见解。 显示英文摘要

摘要： Diffusion models have emerged as a powerful paradigm for modern generative modeling, demonstrating strong potential for large language models (LLMs). Unlike conventional autoregressive (AR) models that generate tokens sequentially, diffusion models enable parallel token sampling, leading to faster generation and eliminating left-to-right generation constraints. Despite their empirical success, the theoretical understanding of diffusion model approaches remains underdeveloped. In this work, we develop convergence guarantees for diffusion language models from an information-theoretic perspective. Our analysis demonstrates that the sampling error, measured by the Kullback-Leibler (KL) divergence, decays inversely with the number of iterations $T$ and scales linearly with the mutual information between tokens in the target text sequence. In particular, we establish matching upper and lower bounds, up to some constant factor, to demonstrate the tightness of our convergence analysis. These results offer novel theoretical insights into the practical effectiveness of diffusion language models.