Title: Multi-Constraint Safe Reinforcement Learning via Closed-form Solution for Log-Sum-Exp Approximation of Control Barrier Functions Show Chinese title

Title: 多约束安全强化学习通过控制屏障函数的Log-Sum-Exp近似闭式解实现

Abstract: The safety of training task policies and their subsequent application using reinforcement learning (RL) methods has become a focal point in the field of safe RL. A central challenge in this area remains the establishment of theoretical guarantees for safety during both the learning and deployment processes. Given the successful implementation of Control Barrier Function (CBF)-based safety strategies in a range of control-affine robotic systems, CBF-based safe RL demonstrates significant promise for practical applications in real-world scenarios. However, integrating these two approaches presents several challenges. First, embedding safety optimization within the RL training pipeline requires that the optimization outputs be differentiable with respect to the input parameters, a condition commonly referred to as differentiable optimization, which is non-trivial to solve. Second, the differentiable optimization framework confronts significant efficiency issues, especially when dealing with multi-constraint problems. To address these challenges, this paper presents a CBF-based safe RL architecture that effectively mitigates the issues outlined above. The proposed approach constructs a continuous AND logic approximation for the multiple constraints using a single composite CBF. By leveraging this approximation, a close-form solution of the quadratic programming is derived for the policy network in RL, thereby circumventing the need for differentiable optimization within the end-to-end safe RL pipeline. This strategy significantly reduces computational complexity because of the closed-form solution while maintaining safety guarantees. Simulation results demonstrate that, in comparison to existing approaches relying on differentiable optimization, the proposed method significantly reduces training computational costs while ensuring provable safety throughout the training process. Show Chinese abstract

Abstract: 训练任务策略的安全性及其后续使用强化学习（RL）方法的应用已成为安全RL领域的一个焦点。该领域中的一个核心挑战仍然是在学习和部署过程中建立理论安全保证。 鉴于基于控制屏障函数（CBF）的安全策略在一系列仿射控制机器人系统中的成功实施，基于CBF的RL展示了在现实场景中实际应用的巨大潜力。然而，将这两种方法结合起来提出了几个挑战。 首先，将安全优化嵌入到RL训练管道中要求优化输出相对于输入参数可微，这通常被称为可微优化，解决起来并不简单。 其次，可微优化框架面临显著的效率问题，尤其是在处理多约束问题时。 为了解决这些挑战，本文提出了一种基于CBF的RL架构，有效地缓解了上述问题。 所提出的方案通过单一复合CBF构建多个约束的连续AND逻辑近似。通过利用这一近似，推导出了RL中策略网络的二次规划的闭式解，从而避免了在端到端安全RL管道中需要可微优化。 由于闭式解的存在，这种策略显著降低了计算复杂度，同时保持了安全保证。 仿真结果显示，与依赖可微优化的现有方法相比，所提出的方法显著减少了训练计算成本，并在整个训练过程中确保了可证明的安全性。