标题： 关于RLVR的优化动力学：梯度差距和步长阈值 显示英文标题

标题： On the optimization dynamics of RLVR: Gradient gap and step size thresholds

摘要： 基于可验证奖励的强化学习（RLVR），它使用简单的二进制反馈来微调大型语言模型，已经显示出显著的实证成功。 然而，对其为何有效缺乏系统的理解。 本文通过在完整响应（轨迹）和标记层面分析其训练过程，为RLVR建立了理论基础。 我们分析的核心是一个称为梯度差距的量，它形式化了从低奖励到高奖励响应空间区域的改进方向。 我们证明，收敛关键取决于将更新方向与这个梯度差距对齐。 此外，我们根据梯度差距的大小推导出一个精确的步长阈值：低于该阈值时，学习收敛，而高于该阈值时，性能会崩溃。 我们的理论进一步预测了关键步长必须如何随着响应长度和成功率进行缩放，从而解释了为什么实际启发式方法如长度归一化可以提高稳定性，并表明在固定学习率下，成功率会在严格低于$100\%$的情况下停滞。 我们通过受控赌博机模拟验证了这些预测。 显示英文摘要

摘要： Reinforcement Learning with Verifiable Rewards (RLVR), which uses simple binary feedback to post-train large language models, has shown significant empirical success. However, a principled understanding of why it works has been lacking. This paper builds a theoretical foundation for RLVR by analyzing its training process at both the full-response (trajectory) and token levels. Central to our analysis is a quantity called the Gradient Gap, which formalizes the direction of improvement from low-reward to high-reward regions of the response space. We prove that convergence critically depends on aligning the update direction with this Gradient Gap. Moreover, we derive a sharp step-size threshold based on the magnitude of the Gradient Gap: below it, learning converges, whereas above it, performance collapses. Our theory further predicts how the critical step size must scale with response length and the success rate, thereby explaining why practical heuristics such as length normalization improve stability and showing that, with a fixed learning rate, the success rate can stagnate strictly below $100\%$. We validate these predictions through controlled bandit simulations.