标题： 基于难度的偏好数据选择通过DPO隐式奖励差距 显示英文标题

标题： Difficulty-Based Preference Data Selection by DPO Implicit Reward Gap

摘要： 对齐大型语言模型（LLMs）与人类偏好是人工智能研究中的一个关键挑战。 虽然像从人类反馈中进行强化学习（RLHF）和直接偏好优化（DPO）这样的方法被广泛使用，但它们通常依赖于大型且昂贵的偏好数据集。 当前的工作缺乏专门针对偏好数据的高质量数据选择方法。 在本工作中，我们引入了一种基于难度的偏好数据集数据选择策略，该策略建立在DPO隐式奖励机制的基础上。 通过选择具有较小DPO隐式奖励差距的偏好数据示例，这些示例表明更具挑战性的案例，我们提高了数据效率和模型对齐度。 我们的方法在多个数据集和对齐任务中始终优于五个强大的基线，仅使用原始数据的10%就实现了卓越的性能。 这种有原则的高效选择方法为在有限资源下扩展LLM对齐提供了一个有前景的解决方案。 显示英文摘要

摘要： Aligning large language models (LLMs) with human preferences is a critical challenge in AI research. While methods like Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) are widely used, they often rely on large, costly preference datasets. The current work lacks methods for high-quality data selection specifically for preference data. In this work, we introduce a novel difficulty-based data selection strategy for preference datasets, grounded in the DPO implicit reward mechanism. By selecting preference data examples with smaller DPO implicit reward gaps, which are indicative of more challenging cases, we improve data efficiency and model alignment. Our approach consistently outperforms five strong baselines across multiple datasets and alignment tasks, achieving superior performance with only 10\% of the original data. This principled, efficient selection method offers a promising solution for scaling LLM alignment with limited resources.