标题： 增量多智能体玻尔兹曼 Q 学习的确定性模型：瞬时合作、亚稳态和振荡 显示英文标题

标题： Deterministic Model of Incremental Multi-Agent Boltzmann Q-Learning: Transient Cooperation, Metastability, and Oscillations

摘要： 多智能体强化学习涉及在共享环境中共同学习的智能体，导致出现对初始条件和参数变化敏感的涌现动力学。 一种动力系统方法，用于研究多组分系统随时间演化的特性，通过构建随机算法的确定性近似模型，揭示了一些潜在的动力学。 在本工作中，我们证明即使在独立Q学习与玻尔兹曼探索策略的最简单情况下，实际算法与之前近似之间也存在显著差异。 我们详细说明了为什么这些模型实际上近似的是有趣的变体，而不是原始的增量算法。 为了解释这些差异，我们引入了一个新的离散时间近似模型，该模型明确考虑了学习过程中智能体的更新频率，并表明其动力学与先前模型的简化动力学有根本的不同。 我们通过将其应用于社会困境中的自发合作问题，特别是以囚徒困境作为最简单的案例研究，来展示我们方法的实用性。 我们确定了从外部视角来看学习行为表现为长期稳定合作的条件。 然而，我们的模型显示，这种行为仅仅是亚稳态的瞬态阶段，并非真正的均衡状态，因此容易被利用。 我们进一步举例说明了特定的参数设置如何显著加剧独立学习中的动态目标问题。 通过对我们模型的系统分析，我们展示了增加折扣因子会引发振荡，从而阻止收敛到联合策略。 这些振荡来源于超临界尼马克-萨克尔分岔，它将唯一的稳定固定点转化为被稳定极限环包围的不稳定焦点。 显示英文摘要

摘要： Multi-Agent Reinforcement Learning involves agents that learn together in a shared environment, leading to emergent dynamics sensitive to initial conditions and parameter variations. A Dynamical Systems approach, which studies the evolution of multi-component systems over time, has uncovered some of the underlying dynamics by constructing deterministic approximation models of stochastic algorithms. In this work, we demonstrate that even in the simplest case of independent Q-learning with a Boltzmann exploration policy, significant discrepancies arise between the actual algorithm and previous approximations. We elaborate why these models actually approximate interesting variants rather than the original incremental algorithm. To explain the discrepancies, we introduce a new discrete-time approximation model that explicitly accounts for agents' update frequencies within the learning process and show that its dynamics fundamentally differ from the simplified dynamics of prior models. We illustrate the usefulness of our approach by applying it to the question of spontaneous cooperation in social dilemmas, specifically the Prisoner's Dilemma as the simplest case study. We identify conditions under which the learning behaviour appears as long-term stable cooperation from an external perspective. However, our model shows that this behaviour is merely a metastable transient phase and not a true equilibrium, making it exploitable. We further exemplify how specific parameter settings can significantly exacerbate the moving target problem in independent learning. Through a systematic analysis of our model, we show that increasing the discount factor induces oscillations, preventing convergence to a joint policy. These oscillations arise from a supercritical Neimark-Sacker bifurcation, which transforms the unique stable fixed point into an unstable focus surrounded by a stable limit cycle.