标题： Phi-4-Mini-Reasoning：探究小型推理语言模型在数学中的极限 显示英文标题

标题： Phi-4-Mini-Reasoning: Exploring the Limits of Small Reasoning Language Models in Math

摘要： Chain-of-Thought (CoT) 显著增强了大型语言模型 (LLMs) 的形式推理能力，通过训练它们明确生成中间推理步骤。 尽管 LLMs 很容易从这些技术中受益，但由于小规模语言模型 (SLMs) 的容量限制，提高它们的推理能力仍然具有挑战性。 Deepseek-R1 的最新研究表明，从 LLM 生成的合成数据中蒸馏可以显著提高 SLM 的推理能力。 然而，详细的建模方法没有被披露。 在这项工作中，我们提出了一个系统性的训练方法，适用于 SLM，该方法包括四个步骤：(1) 在多样化的蒸馏长 CoT 数据上的大规模中期训练，(2) 在高质量长 CoT 数据上的监督微调，(3) 利用精心策划的偏好数据集进行 Rollout DPO，(4) 带有可验证奖励的强化学习 (RL)。 我们将这种方法应用于 Phi-4-Mini，这是一个紧凑型的 38 亿参数模型。 由此产生的 Phi-4-Mini-Reasoning 模型在数学推理任务上超过了更大的推理模型，例如，在 Math-500 上比 DeepSeek-R1-Distill-Qwen-7B 高出 3.2 分，比 DeepSeek-R1-Distill-Llama-8B 高出 7.7 分。 我们的结果显示，一个经过精心设计的训练方法，使用大规模高质量的 CoT 数据，即使在资源受限的小模型中也能有效解锁强大的推理能力。 显示英文摘要

摘要： Chain-of-Thought (CoT) significantly enhances formal reasoning capabilities in Large Language Models (LLMs) by training them to explicitly generate intermediate reasoning steps. While LLMs readily benefit from such techniques, improving reasoning in Small Language Models (SLMs) remains challenging due to their limited model capacity. Recent work by Deepseek-R1 demonstrates that distillation from LLM-generated synthetic data can substantially improve the reasoning ability of SLM. However, the detailed modeling recipe is not disclosed. In this work, we present a systematic training recipe for SLMs that consists of four steps: (1) large-scale mid-training on diverse distilled long-CoT data, (2) supervised fine-tuning on high-quality long-CoT data, (3) Rollout DPO leveraging a carefully curated preference dataset, and (4) Reinforcement Learning (RL) with Verifiable Reward. We apply our method on Phi-4-Mini, a compact 3.8B-parameter model. The resulting Phi-4-Mini-Reasoning model exceeds, on math reasoning tasks, much larger reasoning models, e.g., outperforming DeepSeek-R1-Distill-Qwen-7B by 3.2 points and DeepSeek-R1-Distill-Llama-8B by 7.7 points on Math-500. Our results validate that a carefully designed training recipe, with large-scale high-quality CoT data, is effective to unlock strong reasoning capabilities even in resource-constrained small models.