标题： 一种对与*PO的人类对齐的实际分析 显示英文标题

标题： A Practical Analysis of Human Alignment with *PO

摘要： 在最先进的人员对齐方法中，偏好优化方法（*PO）处于前沿。 先前的研究通常集中在确定表现最好的方法，通常涉及超参数的网格搜索，这对普通从业者来说可能不切实际。 在本文中，我们研究了现有最先进的方法在现实的分布外（OOD）场景中对不同超参数的鲁棒性，该场景反映了人员对齐的实际应用。 我们的目标是通过各种指标（如KL散度和响应长度）的经验方法，找到能通过不同指标增加获得更好结果可能性的方法。 我们还引入了LN-DPO，这是DPO的一个简单长度归一化版本，在超参数上更稳定，有效减少了平均响应长度，并提高了性能。 我们对最先进的无参考（即SimPO）和有参考（即DPO和LN-DPO）方法的分析表明，它们在其最佳状态（即最佳可能情况）下表现相似。 然而，我们发现当偏离最佳可能情况时，性能的变化模式有很大差异。 显示英文摘要

摘要： At the forefront of state-of-the-art human alignment methods are preference optimization methods (*PO). Prior research has often concentrated on identifying the best-performing method, typically involving a grid search over hyperparameters, which can be impractical for general practitioners. In this paper, we examine the robustness of existing state-of-the-art methods to varying hyperparameters in a realistic out-of-distribution (OOD) scenario that mirrors real-world applications of human alignment. Our goal is to empirically find the method that increases the likelihood of achieving better results through the lens of various metrics, such as KL divergence and response length. We also introduce LN-DPO, a simple length-normalized version of DPO that is more stable across hyperparameters, effectively reduces the average response length, and improves performance. Our analysis of state-of-the-art reference-free (i.e., SimPO) and reference-dependent (i.e., DPO and LN-DPO) methods reveals that they perform similarly at their peak (i.e., best possible scenario). However, we uncover that the pattern of change in performance greatly varies as we move away from the best possible scenario.