标题： 自推测掩码扩散 显示英文标题

标题： Self-Speculative Masked Diffusions

摘要： 我们提出自推测掩码扩散模型，这是一种新的掩码扩散生成模型，适用于离散数据，生成样本所需的函数评估次数显著减少。 标准的掩码扩散模型在当前掩码位置上预测分解的对数几率。 然后会采样一定数量的掩码位置，然而，分解近似意味着一次采样太多位置会导致样本质量下降。 因此，为了生成高质量的数据，需要许多模拟步骤和神经网络函数评估。 我们通过在掩码位置上生成非分解预测来减少计算负担。 这是通过将最终的Transformer注意力掩码从非因果改为因果实现的，从而通过一种新颖的、模型集成的推测采样机制实现草稿标记生成和并行验证。 这使得在一次前向传递中得到掩码位置上的非分解预测分布。 我们将该方法应用于GPT2规模的文本建模和蛋白质序列生成，发现与标准掩码扩散模型相比，所需网络前向传递次数减少了约2倍。 显示英文摘要

摘要： We present self-speculative masked diffusions, a new class of masked diffusion generative models for discrete data that require significantly fewer function evaluations to generate samples. Standard masked diffusion models predict factorized logits over currently masked positions. A number of masked positions are then sampled, however, the factorization approximation means that sampling too many positions in one go leads to poor sample quality. As a result, many simulation steps and therefore neural network function evaluations are required to generate high-quality data. We reduce the computational burden by generating non-factorized predictions over masked positions. This is achieved by modifying the final transformer attention mask from non-causal to causal, enabling draft token generation and parallel validation via a novel, model-integrated speculative sampling mechanism. This results in a non-factorized predictive distribution over masked positions in a single forward pass. We apply our method to GPT2 scale text modelling and protein sequences generation, finding that we can achieve a ~2x reduction in the required number of network forward passes relative to standard masked diffusion models.