标题： 针对平均场控制博弈的高效可扩展深度强化学习 显示英文标题

标题： Efficient and Scalable Deep Reinforcement Learning for Mean Field Control Games

摘要： 平均场控制博弈（MFCGs）提供了一个强大的理论框架，用于分析无限多相互作用代理的系统，融合了平均场博弈（MFGs）和平均场控制（MFC）的元素。然而，求解表征MFCG平衡状态的耦合Hamilton-Jacobi-Bellman方程和Fokker-Planck方程仍然是一个重要的计算挑战，特别是在高维或复杂环境中。 本文提出了一种可扩展的深度强化学习（RL）方法来近似MFCGs的平衡解。在此前工作的基础上，我们将无限代理随机控制问题重新表述为马尔可夫决策过程，在此过程中每个代表性代理与演化的均值场分布交互。我们以先前论文中的演员-评论家算法（Angiuli等人，2024）为基础，并提出了几个版本的更具可扩展性和效率更高的算法，利用了包括并行样本收集（批处理）、小批量处理、目标网络、近端策略优化（PPO）、广义优势估计（GAE）和熵正则化等技术。 通过利用这些技术，我们有效地提高了基线算法的效率、可扩展性和训练稳定性。我们在一个具有解析解的线性二次基准问题上评估了我们的方法。结果表明，我们提出的某些方法实现了更快的收敛速度，并且接近理论最优解，相比基线算法在样本效率方面高出一个数量级。 我们的工作为适应深度RL解决更复杂的MFCGs奠定了基础，例如大规模自主交通系统、多企业经济竞争以及银行间借贷等问题。 显示英文摘要

摘要： Mean Field Control Games (MFCGs) provide a powerful theoretical framework for analyzing systems of infinitely many interacting agents, blending elements from Mean Field Games (MFGs) and Mean Field Control (MFC). However, solving the coupled Hamilton-Jacobi-Bellman and Fokker-Planck equations that characterize MFCG equilibria remains a significant computational challenge, particularly in high-dimensional or complex environments. This paper presents a scalable deep Reinforcement Learning (RL) approach to approximate equilibrium solutions of MFCGs. Building on previous works, We reformulate the infinite-agent stochastic control problem as a Markov Decision Process, where each representative agent interacts with the evolving mean field distribution. We use the actor-critic based algorithm from a previous paper (Angiuli et.al., 2024) as the baseline and propose several versions of more scalable and efficient algorithms, utilizing techniques including parallel sample collection (batching); mini-batching; target network; proximal policy optimization (PPO); generalized advantage estimation (GAE); and entropy regularization. By leveraging these techniques, we effectively improved the efficiency, scalability, and training stability of the baseline algorithm. We evaluate our method on a linear-quadratic benchmark problem, where an analytical solution to the MFCG equilibrium is available. Our results show that some versions of our proposed approach achieve faster convergence and closely approximate the theoretical optimum, outperforming the baseline algorithm by an order of magnitude in sample efficiency. Our work lays the foundation for adapting deep RL to solve more complicated MFCGs closely related to real life, such as large-scale autonomous transportation systems, multi-firm economic competition, and inter-bank borrowing problems.