标题： ADPO：基于锚点的直接偏好优化 显示英文标题

标题： ADPO: Anchored Direct Preference Optimization

摘要： 直接偏好优化（DPO）在标注器噪声和分布偏移下效果不佳，因为它基于硬性成对标签，并且仅对对数概率差异进行正则化。我们引入了锚定直接偏好优化（ADPO），这是一种通过参考锚定扩展偏好学习到软列表监督的框架。ADPO最小化KL(q || softmax((s - s_ref) / tau_anc))，这（i）通过适当选择目标q、锚定策略和温度，恢复了监督微调、知识蒸馏、最大熵强化学习和DPO作为特殊情况；（ii）由softmax费舍尔度量引导了一个隐式的信任区域，与锚定无关；并且（iii）支持稳定的动态锚定更新。经验上，我们观察到任务相关的权衡：动态锚定在噪声下改善在线探索，而固定锚定在离线蒸馏中表现更优，在我们的基准测试中，学生-教师KL减少了最多170到5000倍。 显示英文摘要

摘要： Direct Preference Optimization (DPO) is effective but brittle under annotator noise and distribution shift because it operates on hard, pairwise labels and only regularizes log-probability differences. We introduce Anchored Direct Preference Optimization (ADPO), a framework that extends preference learning to soft listwise supervision via reference anchoring. ADPO minimizes KL(q || softmax((s - s_ref) / tau_anc)), which (i) recovers supervised fine-tuning, knowledge distillation, maximum-entropy reinforcement learning, and DPO as special cases through suitable choices of target q, anchor policy, and temperature; (ii) induces an implicit trust region governed by the softmax Fisher metric, independent of the anchor; and (iii) supports stable dynamic-anchor updates. Empirically, we observe a task-dependent tradeoff: dynamic anchors improve online exploration under noise, while fixed anchors excel at offline distillation, achieving up to 170 to 5000 times reduction in student-teacher KL on our benchmarks.