Title: Game On: Towards Language Models as RL Experimenters Show Chinese title

Title: 游戏开始：迈向作为强化学习实验者的语言模型

Abstract: We propose an agent architecture that automates parts of the common reinforcement learning experiment workflow, to enable automated mastery of control domains for embodied agents. To do so, it leverages a VLM to perform some of the capabilities normally required of a human experimenter, including the monitoring and analysis of experiment progress, the proposition of new tasks based on past successes and failures of the agent, decomposing tasks into a sequence of subtasks (skills), and retrieval of the skill to execute - enabling our system to build automated curricula for learning. We believe this is one of the first proposals for a system that leverages a VLM throughout the full experiment cycle of reinforcement learning. We provide a first prototype of this system, and examine the feasibility of current models and techniques for the desired level of automation. For this, we use a standard Gemini model, without additional fine-tuning, to provide a curriculum of skills to a language-conditioned Actor-Critic algorithm, in order to steer data collection so as to aid learning new skills. Data collected in this way is shown to be useful for learning and iteratively improving control policies in a robotics domain. Additional examination of the ability of the system to build a growing library of skills, and to judge the progress of the training of those skills, also shows promising results, suggesting that the proposed architecture provides a potential recipe for fully automated mastery of tasks and domains for embodied agents. Show Chinese abstract

Abstract: 我们提出了一种代理架构，该架构自动化了常见强化学习实验工作流程的部分内容，以实现对具身代理的控制领域的自动化掌握。 为此，它利用视觉语言模型（VLM）执行通常需要人类实验者完成的一些能力，包括监控和分析实验进展、基于代理过去的成功与失败提出新任务、将任务分解为子任务序列（技能），以及检索要执行的技能——从而使我们的系统能够为学习构建自动化的课程。 我们认为，这是首次提出的一种系统性利用VLM贯穿强化学习整个实验周期的方案之一。 我们提供了该系统的首个原型，并研究了当前模型和技术在实现所需自动化水平方面的可行性。 为此，我们使用一个标准的Gemini模型（未进行额外微调），向条件化Actor-Critic算法提供技能课程，以引导数据收集，从而帮助学习新技能。 通过这种方式收集的数据被证明对于学习和迭代改进机器人领域的控制策略是有用的。 系统构建技能库的能力以及评估这些技能训练进展的进一步考察也显示出有希望的结果，这表明所提出的架构为具身代理的任务和领域自动化掌握提供了潜在的解决方案。