标题： 迷你-N内部采样最佳：一种上下文质量奖励模型用于可靠且高效的最佳-N采样 显示英文标题

标题： Best of mini-N in-loop Sampling: A Contextual Quality Reward Model for Reliable and Efficient Best-of-N Sampling

摘要： 现代偏好对齐技术，如最佳- N (BoN) 采样，依赖于通过成对比较数据训练的奖励模型。 虽然在学习相对偏好方面有效，但这种范式无法捕捉响应可接受性的信号，使系统容易选择许多不可接受选项中相对较坏的一个。 这对于困难提示尤其成问题，因为在这样的提示中，这种错误接受的风险会随着样本数量的增加而增加。 在本文中，我们通过引入一种新的数据收集和建模框架来解决这一关键可靠性差距。 通过借鉴离散选择模型，用一个外部选项扩充偏好数据，我们训练了一个奖励模型，不仅可以区分什么是更好的，还可以区分什么是足够好的。 我们利用这种能力创建了一种自适应推理策略，即最小- N 内循环最佳选择，该策略将生成预算划分为带有校准的早期退出条件的顺序循环。 我们的实验表明，当作为对齐护栏进行调整时，它将可靠性故障减少了70%，当作为推理加速器进行调整时，在IMDB情感设置中平均推理速度提高了超过22%。 因此，我们为从业者提供了一个有原则且灵活的框架，以显式管理可靠性与计算效率之间的权衡。 显示英文摘要

摘要： Modern preference alignment techniques, such as Best-of-N (BoN) sampling, rely on reward models trained with pairwise comparison data. While effective at learning relative preferences, this paradigm fails to capture a signal of response acceptability, leaving systems vulnerable to selecting the least bad of many unacceptable options. This is particularly problematic for hard prompts, where the risk of such false acceptances increases with the number of samples. In this paper, we address this critical reliability gap by introducing a new data collection and modeling framework. By augmenting preference data with an outside option, inspired by discrete choice models, we train a reward model that can distinguish not just what is better, but what is good enough. We leverage this capability to create an adaptive inference strategy, best of mini-N in-loop, which partitions the generation budget into sequential loops with a calibrated, early-exit condition. Our experiments show that when tuned as an alignment guardrail, it reduces reliability failures by 70%, and when tuned as an inference accelerator, it improves average inference speed by over 22% in IMDB-sentiment setting. We thus provide a principled and flexible framework for practitioners to explicitly manage the trade-off between reliability and computational efficiency.