标题： 3D-属性：识别DPO中的挑战并规划前进路径 显示英文标题

标题： 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward

摘要： 将大型语言模型（LLMs）与人类偏好对齐已引起广泛关注，其中近端策略优化（PPO）是一种标准但计算成本较高的方法，而直接偏好优化（DPO）则是一种更高效的替代方法。 虽然DPO具有简单性，但它在最先进的LLMs中仍未被充分利用，这表明可能存在潜在的局限性。 在本工作中，我们重新审视DPO，分析其理论基础和实验性能以填补这一差距。 我们确定了三个关键特性，称为3D特性，这些特性源自DPO的学习过程：拒绝响应可能性的急剧下降、退化为响应抑制以及对未见过的响应的分散效应。 我们表明，这些问题源于DPO的优化动态，其中选择和拒绝响应梯度之间的相互作用导致了不稳定性。 我们的发现得到了在受控玩具模型和实际LLM任务（包括数学问题解决和指令遵循）上的实验支持。 为了解决这些挑战，我们提出了简单的正则化技术，以提高训练稳定性和性能。 此外，我们研究了偏好数据分布如何影响DPO的有效性，提供了关于对齐模型如何处理域外（OOD）数据的见解。 我们的工作将这些观察结果与更广泛的研究联系起来，并为DPO的局限性提供了理论解释。 我们希望这些见解能指导未来无奖励模型偏好学习的发展，使其更接近基于奖励模型的方法。 显示英文摘要

摘要： Aligning large language models (LLMs) with human preferences has gained significant attention, with Proximal Policy Optimization (PPO) as a standard yet computationally expensive method and Direct Preference Optimization (DPO) as a more efficient alternative. While DPO offers simplicity, it remains underutilized in state-of-the-art LLMs, suggesting potential limitations. In this work, we revisit DPO, analyzing its theoretical foundations and empirical performance to bridge this gap. We identify three key properties, termed 3D properties, that emerge from DPO's learning process: Drastic drop in rejected response likelihood, Degradation into response suppression, and Dispersion effect on unseen responses. We show that these issues arise from DPO's optimization dynamics, where the interaction between chosen and rejected response gradients leads to instability. Our findings are supported by experiments on both a controlled toy model and real-world LLM tasks, including mathematical problem-solving and instruction following. To address these challenges, we propose simple regularization techniques that improve training stability and performance. Additionally, we examine how preference data distribution impacts DPO's effectiveness, offering insights into how alignment models handle out-of-domain (OOD) data. Our work connects these observations to broader research and provides a theoretical explanation for DPO's limitations. We hope these insights will guide future advancements in reward-model-free preference learning, bringing it closer to reward-model-based approaches.