标题： 通过在线世界模型进行规划的持续强化学习 显示英文标题

标题： Continual Reinforcement Learning by Planning with Online World Models

摘要： 持续强化学习（CRL）指的是一个自然情境，其中智能体需要通过不断试错来持续进化，以解决依次呈现的多个任务。 CRL最大的障碍之一是当学习新任务时，智能体可能会忘记如何解决之前的任务，这被称为灾难性遗忘。 在本文中，我们提出通过使用在线世界模型来进行规划来解决这一挑战。 具体而言，我们在线学习一个跟随领导者浅层模型来捕捉世界动态，在此模型中，我们使用模型预测控制来解决由任何奖励函数指定的一组任务。 在温和假设下，该在线世界模型具有证明的遗憾界$\mathcal{O}(\sqrt{K^2D\log(T)})$，因此在构造上对遗忘免疫。 规划器仅基于最新的在线模型搜索动作，从而形成一个FTRL在线代理（OA），该代理逐步更新。 为了评估OA，我们进一步设计了Continual Bench，这是一个专门用于CRL的环境，并在相同的模型规划算法框架下与多个强基线进行比较。 实验结果表明，OA能够持续学习以解决新任务而不遗忘旧技能，其性能优于使用各种持续学习技术的深度世界模型构建的代理。 显示英文摘要

摘要： Continual reinforcement learning (CRL) refers to a naturalistic setting where an agent needs to endlessly evolve, by trial and error, to solve multiple tasks that are presented sequentially. One of the largest obstacles to CRL is that the agent may forget how to solve previous tasks when learning a new task, known as catastrophic forgetting. In this paper, we propose to address this challenge by planning with online world models. Specifically, we learn a Follow-The-Leader shallow model online to capture the world dynamics, in which we plan using model predictive control to solve a set of tasks specified by any reward functions. The online world model is immune to forgetting by construction with a proven regret bound of $\mathcal{O}(\sqrt{K^2D\log(T)})$ under mild assumptions. The planner searches actions solely based on the latest online model, thus forming a FTL Online Agent (OA) that updates incrementally. To assess OA, we further design Continual Bench, a dedicated environment for CRL, and compare with several strong baselines under the same model-planning algorithmic framework. The empirical results show that OA learns continuously to solve new tasks while not forgetting old skills, outperforming agents built on deep world models with various continual learning techniques.