标题： wd1：扩散语言模型推理中的加权策略优化 显示英文标题

标题： wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models

摘要： 通过强化学习（RL）提高基于扩散的大语言模型（dLLMs）的推理能力仍然是一个开放性问题。dLLMs似然函数的不可处理性要求在每次策略优化步骤中近似当前策略、旧策略和参考策略的似然。这种依赖性引入了额外的计算开销，并可能导致较大的偏差——特别是在用于重要性采样的策略比率分母中出现近似误差时。为缓解这些问题，我们引入了$\mathtt{wd1}$，一种新的策略优化方法，将目标重新表述为加权似然，仅需对当前参数化策略似然进行一次近似。在广泛使用的推理基准上的实验表明，$\mathtt{wd1}$在无需监督微调（SFT）或任何监督数据的情况下，优于现有的dLLMs RL方法，准确率最高提高了16%。$\mathtt{wd1}$带来了额外的计算优势，包括减少训练时间和每梯度步骤的函数评估次数（NFEs）。这些发现结合该方法实现的简单性和R1-Zero类似训练（无SFT），使$\mathtt{wd1}$成为应用于dLLMs推理的更有效和高效的RL方法。 显示英文摘要

摘要： Improving the reasoning capabilities of diffusion-based large language models (dLLMs) through reinforcement learning (RL) remains an open problem. The intractability of dLLMs likelihood function necessitates approximating the current, old, and reference policy likelihoods at each policy optimization step. This reliance introduces additional computational overhead and lead to potentially large bias -- particularly when approximation errors occur in the denominator of policy ratios used for importance sampling. To mitigate these issues, we introduce $\mathtt{wd1}$, a novel policy optimization approach that reformulates the objective as a weighted likelihood, requiring only a single approximation for the current parametrized policy likelihood. Experiments on widely used reasoning benchmarks demonstrate that $\mathtt{wd1}$, without supervised fine-tuning (SFT) or any supervised data, outperforms existing RL methods for dLLMs, achieving up to 16% higher accuracy. $\mathtt{wd1}$ delivers additional computational gains, including reduced training time and fewer function evaluations (NFEs) per gradient step. These findings, combined with the simplicity of method's implementation and R1-Zero-like training (no SFT), position $\mathtt{wd1}$ as a more effective and efficient method for applying RL to dLLMs reasoning.