标题： 一种拉格朗日方法在安全强化学习中的实证研究 显示英文标题

标题： An Empirical Study of Lagrangian Methods in Safe Reinforcement Learning

摘要： 在安全关键领域，如机器人技术、导航和电力系统中，会出现需要平衡性能与相关约束的约束优化问题。 安全强化学习提供了一个框架来解决这些挑战，其中拉格朗日方法是一种常用的选择。 然而，拉格朗日方法的有效性在很大程度上取决于拉格朗日乘子$\lambda$的选择，该乘子控制回报与约束成本之间的权衡。 一种常见方法是在训练过程中自动更新该乘子。 尽管这在实践中是标准的，但关于自动更新的鲁棒性及其对整体性能的影响，仍缺乏充分的实证证据。 因此，我们分析了（i）安全强化学习中拉格朗日乘子的最优性以及（ii）其稳定性，涵盖多种任务。 我们提供了$\lambda$-轮廓图，这些图完整地展示了优化问题中回报与约束成本之间的权衡。 这些轮廓图显示了$\lambda$的高度敏感性，并进一步确认了在选择最优值$\lambda^*$时缺乏普遍的直觉。 我们的研究结果还表明，由于它们的学习轨迹存在巨大差异，自动乘子更新能够恢复甚至有时超过在$\lambda^*$处找到的最优性能。 此外，我们展示了自动乘子更新在训练过程中表现出振荡行为，这种行为可以通过 PID 控制的更新来缓解。 然而，这种方法需要仔细调整，以在各种任务中实现一致更好的性能。 这突显了在安全强化学习中进一步研究稳定拉格朗日方法的必要性。 用于重现我们结果的代码可在 https://github.com/lindsayspoor/Lagrangian_SafeRL 找到。 显示英文摘要

摘要： In safety-critical domains such as robotics, navigation and power systems, constrained optimization problems arise where maximizing performance must be carefully balanced with associated constraints. Safe reinforcement learning provides a framework to address these challenges, with Lagrangian methods being a popular choice. However, the effectiveness of Lagrangian methods crucially depends on the choice of the Lagrange multiplier $\lambda$, which governs the trade-off between return and constraint cost. A common approach is to update the multiplier automatically during training. Although this is standard in practice, there remains limited empirical evidence on the robustness of an automated update and its influence on overall performance. Therefore, we analyze (i) optimality and (ii) stability of Lagrange multipliers in safe reinforcement learning across a range of tasks. We provide $\lambda$-profiles that give a complete visualization of the trade-off between return and constraint cost of the optimization problem. These profiles show the highly sensitive nature of $\lambda$ and moreover confirm the lack of general intuition for choosing the optimal value $\lambda^*$. Our findings additionally show that automated multiplier updates are able to recover and sometimes even exceed the optimal performance found at $\lambda^*$ due to the vast difference in their learning trajectories. Furthermore, we show that automated multiplier updates exhibit oscillatory behavior during training, which can be mitigated through PID-controlled updates. However, this method requires careful tuning to achieve consistently better performance across tasks. This highlights the need for further research on stabilizing Lagrangian methods in safe reinforcement learning. The code used to reproduce our results can be found at https://github.com/lindsayspoor/Lagrangian_SafeRL.