Title: KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning Show Chinese title

Title: KDRL：通过统一的知识蒸馏和强化学习对推理大模型进行后训练

Abstract: Recent advances in large language model (LLM) post-training have leveraged two distinct paradigms to enhance reasoning capabilities: reinforcement learning (RL) and knowledge distillation (KD). While RL enables the emergence of complex reasoning behaviors, it often suffers from low sample efficiency when the initial policy struggles to explore high-reward trajectories. Conversely, KD improves learning efficiency via mimicking the teacher model but tends to generalize poorly to out-of-domain scenarios. In this work, we present \textbf{KDRL}, a \textit{unified post-training framework} that jointly optimizes a reasoning model through teacher supervision (KD) and self-exploration (RL). Specifically, KDRL leverages policy gradient optimization to simultaneously minimize the reverse Kullback-Leibler divergence (RKL) between the student and teacher distributions while maximizing the expected rule-based rewards. We first formulate a unified objective that integrates GRPO and KD, and systematically explore how different KL approximations, KL coefficients, and reward-guided KD strategies affect the overall post-training dynamics and performance. Empirical results on multiple reasoning benchmarks demonstrate that KDRL outperforms GRPO and various KD baselines while achieving a favorable balance between performance and reasoning token efficiency. These findings indicate that integrating KD and RL serves as an effective and efficient strategy to train reasoning LLMs. Show Chinese abstract

Abstract: 大型语言模型（LLM）后训练中的最新进展利用了两种不同的范式来增强推理能力：强化学习（RL）和知识蒸馏（KD）。 虽然 RL 能够促使复杂的推理行为出现，但它常常在初始策略难以探索高奖励轨迹时表现出样本效率低的问题。 相反，KD 通过模仿教师模型来提高学习效率，但往往在处理域外场景时泛化效果不佳。 在这项工作中，我们提出了\textbf{KDRL}，一个联合通过教师监督（KD）和自我探索（RL）来优化推理模型的\textit{统一的后训练框架}。 具体来说，KDRL 利用策略梯度优化同时最小化学生和教师分布之间的反向 Kullback-Leibler 散度（RKL），同时最大化基于规则的期望奖励。 我们首先制定一个统一的目标，将 GRPO 和 KD 集成在一起，并系统地探索不同的 KL 近似方法、KL 系数以及基于奖励的 KD 策略如何影响整体后训练动态和性能。 多项推理基准测试的实证结果显示，KDRL 在性能和推理标记效率之间实现了有利的平衡，同时优于 GRPO 和各种 KD 基线。 这些发现表明，结合 KD 和 RL 是一种有效且高效的训练推理 LLM 的策略。