Title: Distributed Neural Policy Gradient Algorithm for Global Convergence of Networked Multi-Agent Reinforcement Learning Show Chinese title

Title: 网络多智能体强化学习全局收敛的分布式神经策略梯度算法

Abstract: This paper studies the networked multi-agent reinforcement learning (NMARL) problem, where the objective of agents is to collaboratively maximize the discounted average cumulative rewards. Different from the existing methods that suffer from poor expression due to linear function approximation, we propose a distributed neural policy gradient algorithm that features two innovatively designed neural networks, specifically for the approximate Q-functions and policy functions of agents. This distributed neural policy gradient algorithm consists of two key components: the distributed critic step and the decentralized actor step. In the distributed critic step, agents receive the approximate Q-function parameters from their neighboring agents via a time-varying communication networks to collaboratively evaluate the joint policy. In contrast, in the decentralized actor step, each agent updates its local policy parameter solely based on its own approximate Q-function. In the convergence analysis, we first establish the global convergence of agents for the joint policy evaluation in the distributed critic step. Subsequently, we rigorously demonstrate the global convergence of the overall distributed neural policy gradient algorithm with respect to the objective function. Finally, the effectiveness of the proposed algorithm is demonstrated by comparing it with a centralized algorithm through simulation in the robot path planning environment. Show Chinese abstract

Abstract: 本文研究了网络化多智能体强化学习（NMARL）问题，其中智能体的目标是协作最大化折扣后的平均累积奖励。 不同于因线性函数逼近而导致表达能力较差的现有方法，我们提出了一种分布式神经策略梯度算法，该算法特别设计了两个神经网络，分别用于近似智能体的Q函数和策略函数。 这种分布式神经策略梯度算法由两个关键组件组成：分布式批评者步和去中心化行动者步。 在分布式批评者步中，智能体通过随时间变化的通信网络从其邻近智能体接收近似Q函数参数，以协同评估联合策略。 相比之下，在去中心化行动者步中，每个智能体仅基于自身的近似Q函数更新其本地策略参数。 在收敛性分析中，我们首先证明了智能体在分布式批评者步中的联合策略评估具有全局收敛性。 随后，我们严格证明了整体分布式神经策略梯度算法关于目标函数的全局收敛性。 最后，通过在机器人路径规划环境中与集中式算法进行仿真比较，验证了所提出算法的有效性。