标题： 用未标记数据增强离线强化学习 显示英文标题

标题： Augmenting Offline RL with Unlabeled Data

摘要： 最近在离线强化学习（Offline RL）中的进展导致了对基于保守策略更新的方法的更多关注，以解决分布外（OOD）问题。 这些方法通常涉及添加行为正则化或修改评论家学习目标，主要关注数据集中有大量支持的状态或动作。 然而，我们通过断言数据集中不存在的动作或状态并不一定意味着其次优性来挑战这一普遍观点。 在本文中，我们提出了一种新方法来解决OOD问题。 我们引入了一个离线RL的教师-学生框架，并辅以策略相似性度量。 该框架使学生策略不仅能够从离线RL数据集中获得见解，还能从教师策略传递的知识中获益。 教师策略是使用包含状态-动作对的另一个数据集进行训练的，这可以被视为在不直接与环境交互的情况下获得的实际领域知识。 我们认为这种额外的知识是有效解决OOD问题的关键。 这项研究代表了将教师-学生网络集成到actor-critic框架中的重要进展，为离线RL中的知识迁移研究开辟了新途径，并有效解决了OOD挑战。 显示英文摘要

摘要： Recent advancements in offline Reinforcement Learning (Offline RL) have led to an increased focus on methods based on conservative policy updates to address the Out-of-Distribution (OOD) issue. These methods typically involve adding behavior regularization or modifying the critic learning objective, focusing primarily on states or actions with substantial dataset support. However, we challenge this prevailing notion by asserting that the absence of an action or state from a dataset does not necessarily imply its suboptimality. In this paper, we propose a novel approach to tackle the OOD problem. We introduce an offline RL teacher-student framework, complemented by a policy similarity measure. This framework enables the student policy to gain insights not only from the offline RL dataset but also from the knowledge transferred by a teacher policy. The teacher policy is trained using another dataset consisting of state-action pairs, which can be viewed as practical domain knowledge acquired without direct interaction with the environment. We believe this additional knowledge is key to effectively solving the OOD issue. This research represents a significant advancement in integrating a teacher-student network into the actor-critic framework, opening new avenues for studies on knowledge transfer in offline RL and effectively addressing the OOD challenge.