Title: State-Wise Safe Reinforcement Learning With Pixel Observations Show Chinese title

Title: 基于像素观测的状态安全强化学习

Abstract: In the context of safe exploration, Reinforcement Learning (RL) has long grappled with the challenges of balancing the tradeoff between maximizing rewards and minimizing safety violations, particularly in complex environments with contact-rich or non-smooth dynamics, and when dealing with high-dimensional pixel observations. Furthermore, incorporating state-wise safety constraints in the exploration and learning process, where the agent must avoid unsafe regions without prior knowledge, adds another layer of complexity. In this paper, we propose a novel pixel-observation safe RL algorithm that efficiently encodes state-wise safety constraints with unknown hazard regions through a newly introduced latent barrier-like function learning mechanism. As a joint learning framework, our approach begins by constructing a latent dynamics model with low-dimensional latent spaces derived from pixel observations. We then build and learn a latent barrier-like function on top of the latent dynamics and conduct policy optimization simultaneously, thereby improving both safety and the total expected return. Experimental evaluations on the safety-gym benchmark suite demonstrate that our proposed method significantly reduces safety violations throughout the training process, and demonstrates faster safety convergence compared to existing methods while achieving competitive results in reward return. Show Chinese abstract

Abstract: 在安全探索的背景下，强化学习（RL）长期以来一直在平衡最大化奖励和最小化安全违规之间的权衡方面面临挑战，尤其是在具有接触丰富的或非平滑动力学的复杂环境中，以及在处理高维像素观测时。 此外，在探索和学习过程中引入逐状态的安全约束，其中智能体必须在没有先验知识的情况下避免不安全区域，增加了另一层复杂性。 在本文中，我们提出了一种新的基于像素观测的安全强化学习算法，该算法通过一种新引入的潜在障碍函数学习机制，高效地编码未知危险区域的逐状态安全约束。 作为一个联合学习框架，我们的方法首先从像素观测中构建一个低维潜在空间的潜在动力学模型。 然后在潜在动力学之上构建并学习一个潜在障碍函数，并同时进行策略优化，从而提高安全性和总期望回报。 在安全-gym基准套件上的实验评估表明，我们提出的方法在整个训练过程中显著减少了安全违规，并且在获得与现有方法相当的奖励回报的同时，表现出更快的安全收敛。