标题： 预训练策略判别器是通用奖励模型 显示英文标题

标题： Pre-Trained Policy Discriminators are General Reward Models

摘要： 我们通过将奖励建模表述为一个策略判别器，提供了一种新的视角，该判别器量化两个策略之间的差异以生成奖励信号，引导训练策略向具有期望行为的目标策略发展。 基于这一概念性见解，我们提出了一种可扩展的预训练方法，名为策略判别学习（POLAR），该方法训练一个奖励模型（RM）来辨别相同的策略并区分不同的策略。 与依赖绝对偏好的传统奖励建模方法不同，POLAR捕捉一个策略与任意目标策略之间的相对差异，这是一种可扩展的高层次优化目标，适用于建模通用排名关系。 利用POLAR预训练范式，我们提出了一系列参数规模从1.8B到7B的RMs。 实证结果表明，POLAR显著优于传统的非预训练方法，显著提升了RM性能。 例如，与最先进的基线相比，POLAR-7B在STEM任务中将偏好准确率从54.8%提高到81.0%，在创意写作任务中从57.9%提高到85.5%。 POLAR在使用强化微调（RFT）的RLHF中也表现出强大的泛化能力，提供了可靠的奖励信号，并显著提升了策略性能——在20个基准测试中，将LLaMa3.1-8B的平均值从47.36%提高到56.33%，将Qwen2.5-32B从64.49%提高到70.47%。 此外，扩展实验揭示了计算量与性能之间的清晰幂律关系，这由接近0.99的线性相关系数支持。 出色的性能、强大的泛化能力和扩展特性表明，POLAR是开发通用且强大奖励模型的一个有前景的方向。 显示英文摘要

摘要： We offer a novel perspective on reward modeling by formulating it as a policy discriminator, which quantifies the difference between two policies to generate a reward signal, guiding the training policy towards a target policy with desired behaviors. Based on this conceptual insight, we propose a scalable pre-training method named Policy Discriminative Learning (POLAR), which trains a reward model (RM) to discern identical policies and discriminate different ones. Unlike traditional reward modeling methods relying on absolute preferences, POLAR captures the relative difference between one policy and an arbitrary target policy, which is a scalable, high-level optimization objective suitable for modeling generic ranking relationships. Leveraging the POLAR pre-training paradigm, we present a series of RMs with parameter scales from 1.8B to 7B. Empirical results show that POLAR substantially outperforms traditional non-pre-trained methods, significantly enhancing RM performance. For instance, POLAR-7B could improve preference accuracy from 54.8% to 81.0% on STEM tasks and from 57.9% to 85.5% on creative writing tasks compared to SOTA baselines. POLAR also shows robust generalization capabilities in RLHF using Reinforcement Fine-tuning (RFT), providing reliable reward signals and markedly enhancing policy performance--improving LLaMa3.1-8B from an average of 47.36% to 56.33% and Qwen2.5-32B from 64.49% to 70.47% on 20 benchmarks. Moreover, scaling experiments reveal a clear power-law relationship between computation and performance, supported by linear correlation coefficients approaching 0.99. The impressive performance, strong generalization, and scaling properties suggest that POLAR is a promising direction for developing general and strong reward models.