标题： 在线DPO：具有快速-慢速追逐的在线直接偏好优化 显示英文标题

标题： Online DPO: Online Direct Preference Optimization with Fast-Slow Chasing

摘要： 直接偏好优化（DPO）通过直接在人类偏好数据集上进行训练，提高了大型语言模型（LLMs）与人类价值观的一致性，消除了对奖励模型的需求。 然而，由于存在跨领域的用户偏好，直接的持续训练可能导致灾难性遗忘，限制了DPO的性能和效率。 受种内竞争推动物种进化启发，我们提出了一种在线快速-慢速追逐DPO（OFS-DPO）用于偏好对齐，通过模型之间的快速和慢速追逐模拟竞争，以促进快速适应。 具体来说，我们首先推导了在线学习的遗憾上界，并通过最小最大优化模式验证了我们的动机。 基于此，我们引入了两个使用低秩自适应（LoRA）且优化速度不同的相同模块，以模拟种内竞争，并提出了一种新的正则化项来指导它们的学习。 为了进一步缓解跨领域场景中的灾难性遗忘，我们将OFS-DPO扩展为LoRA模块组合策略，从而得到跨领域在线快速-慢速追逐DPO（COFS-DPO）。 该方法利用来自不同任务领域的快速模块参数的线性组合，充分利用历史信息以实现持续的价值对齐。 实验结果表明，OFS-DPO在领域内对齐方面优于DPO，而COFS-DPO在跨领域持续学习场景中表现出色。 显示英文摘要

摘要： Direct Preference Optimization (DPO) improves the alignment of large language models (LLMs) with human values by training directly on human preference datasets, eliminating the need for reward models. However, due to the presence of cross-domain human preferences, direct continual training can lead to catastrophic forgetting, limiting DPO's performance and efficiency. Inspired by intraspecific competition driving species evolution, we propose a Online Fast-Slow chasing DPO (OFS-DPO) for preference alignment, simulating competition through fast and slow chasing among models to facilitate rapid adaptation. Specifically, we first derive the regret upper bound for online learning, validating our motivation with a min-max optimization pattern. Based on this, we introduce two identical modules using Low-rank Adaptive (LoRA) with different optimization speeds to simulate intraspecific competition, and propose a new regularization term to guide their learning. To further mitigate catastrophic forgetting in cross-domain scenarios, we extend the OFS-DPO with LoRA modules combination strategy, resulting in the Cross domain Online Fast-Slow chasing DPO (COFS-DPO). This method leverages linear combinations of fast modules parameters from different task domains, fully utilizing historical information to achive continual value alignment. Experimental results show that OFS-DPO outperforms DPO in in-domain alignment, while COFS-DPO excels in cross-domain continual learning scenarios.