标题： 超越二元奖励：训练语言模型对其不确定性进行推理 显示英文标题

标题： Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty

摘要： 当语言模型（LMs）通过强化学习（RL）进行训练以生成自然语言的“推理链”时，它们在各种困难的问题回答任务上的表现会得到提升。 如今，几乎所有成功的用于推理的强化学习应用都使用二元奖励函数来评估LM输出的正确性。 由于这样的奖励函数不会惩罚猜测或低置信度的输出，它们往往会产生意想不到的副作用，导致校准度下降，并增加LM在其他问题领域生成错误响应（或“幻觉”）的比率。 本文描述了RLCR（带有校准奖励的强化学习），这是一种训练推理模型的方法，能够同时提高准确性和校准的置信度估计。 在RLCR中，LM在推理后生成预测和数值置信度估计。 它们被训练以优化一个奖励函数，该函数将二元正确性分数与Brier分数相结合——这是一种针对置信度估计的评分规则，它激励校准的预测。 我们首先证明，这个奖励函数（或任何使用有界、适当评分规则的类似奖励函数）会产生既准确又校准良好的模型。 接下来，我们展示了在各种数据集上，RLCR在域内和域外评估中显著提高了校准度，而不会损失准确性——优于普通的RL训练和训练为事后分配置信度分数的分类器。 虽然普通RL会损害校准度，但RLCR会提高校准度。 最后，我们证明可以在测试时利用表述的置信度，通过置信度加权缩放方法来提高准确性和校准度。 我们的结果表明，明确优化校准可以产生更普遍可靠的推理模型。 显示英文摘要

摘要： When language models (LMs) are trained via reinforcement learning (RL) to generate natural language "reasoning chains", their performance improves on a variety of difficult question answering tasks. Today, almost all successful applications of RL for reasoning use binary reward functions that evaluate the correctness of LM outputs. Because such reward functions do not penalize guessing or low-confidence outputs, they often have the unintended side-effect of degrading calibration and increasing the rate at which LMs generate incorrect responses (or "hallucinate") in other problem domains. This paper describes RLCR (Reinforcement Learning with Calibration Rewards), an approach to training reasoning models that jointly improves accuracy and calibrated confidence estimation. During RLCR, LMs generate both predictions and numerical confidence estimates after reasoning. They are trained to optimize a reward function that augments a binary correctness score with a Brier score -- a scoring rule for confidence estimates that incentivizes calibrated prediction. We first prove that this reward function (or any analogous reward function that uses a bounded, proper scoring rule) yields models whose predictions are both accurate and well-calibrated. We next show that across diverse datasets, RLCR substantially improves calibration with no loss in accuracy, on both in-domain and out-of-domain evaluations -- outperforming both ordinary RL training and classifiers trained to assign post-hoc confidence scores. While ordinary RL hurts calibration, RLCR improves it. Finally, we demonstrate that verbalized confidence can be leveraged at test time to improve accuracy and calibration via confidence-weighted scaling methods. Our results show that explicitly optimizing for calibration can produce more generally reliable reasoning models.