Title: Tesseract: Tensorised Actors for Multi-Agent Reinforcement Learning Show Chinese title

Title: Tesseract：用于多智能体强化学习的张量化代理

Abstract: Reinforcement Learning in large action spaces is a challenging problem. Cooperative multi-agent reinforcement learning (MARL) exacerbates matters by imposing various constraints on communication and observability. In this work, we consider the fundamental hurdle affecting both value-based and policy-gradient approaches: an exponential blowup of the action space with the number of agents. For value-based methods, it poses challenges in accurately representing the optimal value function. For policy gradient methods, it makes training the critic difficult and exacerbates the problem of the lagging critic. We show that from a learning theory perspective, both problems can be addressed by accurately representing the associated action-value function with a low-complexity hypothesis class. This requires accurately modelling the agent interactions in a sample efficient way. To this end, we propose a novel tensorised formulation of the Bellman equation. This gives rise to our method Tesseract, which views the Q-function as a tensor whose modes correspond to the action spaces of different agents. Algorithms derived from Tesseract decompose the Q-tensor across agents and utilise low-rank tensor approximations to model agent interactions relevant to the task. We provide PAC analysis for Tesseract-based algorithms and highlight their relevance to the class of rich observation MDPs. Empirical results in different domains confirm Tesseract's gains in sample efficiency predicted by the theory. Show Chinese abstract

Abstract: 在大动作空间中的强化学习是一个具有挑战性的问题。 合作多智能体强化学习（MARL）通过在通信和可观测性方面施加各种约束而使问题更加复杂。 在本工作中，我们考虑了影响基于价值的方法和策略梯度方法的根本障碍：随着智能体数量的增加，动作空间呈指数级增长。 对于基于价值的方法，这在准确表示最优价值函数方面带来了挑战。 对于策略梯度方法，这使得训练评论家变得困难，并加剧了评论家滞后的问题。 我们从学习理论的角度表明，通过使用低复杂度假设类准确表示相关动作价值函数，可以解决这两个问题。 这需要以样本高效的方式准确建模智能体之间的交互。 为此，我们提出了贝尔曼方程的一种新颖的张量化公式。 这产生了我们的方法 Tesseract，它将 Q 函数视为一个张量，其模式对应于不同智能体的动作空间。 从 Tesseract 得到的算法在智能体之间分解 Q 张量，并利用低秩张量近似来建模与任务相关的智能体交互。 我们为基于 Tesseract 的算法提供了 PAC 分析，并强调了它们对丰富观察 MDP 类的相关性。 在不同领域的实验结果证实了理论预测的 Tesseract 在样本效率方面的优势。