Title: Robust and Safe Multi-Agent Reinforcement Learning Framework with Communication for Autonomous Vehicles Show Chinese title

Title: 用于自动驾驶车辆的具有通信功能的鲁棒且安全的多智能体强化学习框架

Abstract: Deep multi-agent reinforcement learning (MARL) has been demonstrated effectively in simulations for many multi-robot problems. For autonomous vehicles, the development of vehicle-to-vehicle (V2V) communication technologies provide opportunities to further enhance safety of the system. However, zero-shot transfer of simulator-trained MARL policies to hardware dynamic systems remains challenging, and how to leverage communication and shared information for MARL has limited demonstrations on hardware. This problem is challenged by discrepancies between simulated and physical states, system state and model uncertainties, practical shared information design, and the need for safety guarantees in both simulation and hardware. This paper introduces RSR-RSMARL, a novel Robust and Safe MARL framework that supports Real-Sim-Real (RSR) policy adaptation for multi-agent systems with communication among agents, with both simulation and hardware demonstrations. RSR-RSMARL leverages state (includes shared state information among agents) and action representations considering real system complexities for MARL formulation. The MARL policy is trained with robust MARL algorithm to enable zero-shot transfer to hardware considering the sim-to-real gap. A safety shield module using Control Barrier Functions (CBFs) provides safety guarantee for each individual agent. Experiment results on F1/10th-scale autonomous vehicles with V2V communication demonstrate the ability of RSR-RSMARL framework to enhance driving safety and coordination across multiple configurations. These findings emphasize the importance of jointly designing robust policy representations and modular safety architectures to enable scalable, generalizable RSR transfer in multi-agent autonomy. Show Chinese abstract

Abstract: 多智能体强化学习（MARL）在许多多机器人问题的仿真中已被证明是有效的。对于自动驾驶车辆，车对车（V2V）通信技术的发展为进一步提高系统的安全性提供了机会。然而，从仿真训练的MARL策略到硬件动态系统的零样本迁移仍然具有挑战性，并且如何利用通信和共享信息来进行MARL在硬件上的演示还很有限。该问题受到仿真与物理状态之间的差异、系统状态和模型不确定性、实际共享信息设计以及在仿真和硬件中都需要安全保证等因素的挑战。本文介绍了一种新颖的鲁棒且安全的MARL框架——RSR-RSMARL，它支持带有多智能体之间通信的多智能体系统的实时-仿真-实时（RSR）策略适应，并在仿真和硬件上进行了演示。RSR-RSMARL利用了状态（包括智能体间的共享状态信息）和动作表示，考虑到了真实系统复杂性来构建MARL公式。MARL策略通过鲁棒的MARL算法进行训练，以实现考虑到仿真到现实差距的零样本迁移。使用控制屏障函数（CBF）的安全罩模块为每个单独的智能体提供安全保障。在带有V2V通信的1/10比例自主车辆上的实验结果表明，RSR-RSMARL框架能够增强多种配置下的驾驶安全性和协调性。这些发现强调了联合设计鲁棒策略表示和模块化安全架构的重要性，以实现多智能体自治中的可扩展、通用化的实时-仿真-实时迁移。