标题： 批评性反馈-GRPO：通过自然语言和数值反馈提升LLM推理能力 显示英文标题

标题： Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback

摘要： 数值反馈（如标量奖励）驱动的强化学习 (RL) 最近取得了显著进展，极大地增强了大型语言模型 (LLMs) 的复杂推理能力。 尽管取得了这些成功，我们发现仅使用数值反馈的 RL 面临三个关键挑战：性能瓶颈、自我反思效果有限以及持续失败。 我们进一步表明，即使在表现出性能瓶颈之后，通过利用以批评形式呈现的自然语言反馈，RL 微调模型仍可以对持续失败的问题生成正确的改进。 基于这一洞见，我们提出了 Critique-GRPO，这是一种在线 RL 框架，集成了自然语言和数值反馈以实现有效的策略优化。 Critique-GRPO 使 LLMs 能够同时从初始响应和基于批评的改进中学习，同时保持探索性。 使用 Qwen2.5-7B-Base 和 Qwen3-8B-Base 进行的广泛实验表明， Critique-GRPO 在八个具有挑战性的数学、STEM 和一般推理任务上始终优于基于监督学习和基于 RL 的微调方法，分别提高了约 4.5% 和 5% 的平均 pass@1 分数。 值得注意的是，Critique-GRPO 超过了一个强大的基线，在线 RL 中结合了专家演示。 进一步分析揭示了关于策略探索的两个关键见解：(1) 较高的熵并不总是保证从探索中有效学习；(2) 更长的响应不一定导致更有效的探索。 显示英文摘要

摘要： Recent advances in reinforcement learning (RL) with numerical feedback, such as scalar rewards, have significantly enhanced the complex reasoning capabilities of large language models (LLMs). Despite this success, we identify three key challenges encountered by RL with solely numerical feedback: performance plateaus, limited effectiveness of self-reflection, and persistent failures. We then demonstrate that RL-finetuned models, even after exhibiting performance plateaus, can generate correct refinements on persistently failed problems by leveraging natural language feedback in the form of critiques. Building on this insight, we propose Critique-GRPO, an online RL framework that integrates both natural language and numerical feedback for effective policy optimization. Critique-GRPO enables LLMs to learn from initial responses and critique-guided refinements simultaneously while maintaining exploration. Extensive experiments using Qwen2.5-7B-Base and Qwen3-8B-Base show that Critique-GRPO consistently outperforms supervised learning-based and RL-based fine-tuning approaches across eight challenging mathematical, STEM, and general reasoning tasks, improving average pass@1 scores by approximately 4.5% and 5%, respectively. Notably, Critique-GRPO surpasses a strong baseline that incorporates expert demonstrations within online RL. Further analysis reveals two critical insights about policy exploration: (1) higher entropy does not always guarantee efficient learning from exploration, and (2) longer responses do not necessarily lead to more effective exploration.