Title: Re:Frame -- Retrieving Experience From Associative Memory Show Chinese title

Title: Re:Frame -- 从关联记忆中检索经验

Abstract: Offline reinforcement learning (RL) often deals with suboptimal data when collecting large expert datasets is unavailable or impractical. This limitation makes it difficult for agents to generalize and achieve high performance, as they must learn primarily from imperfect or inconsistent trajectories. A central challenge is therefore how to best leverage scarce expert demonstrations alongside abundant but lower-quality data. We demonstrate that incorporating even a tiny amount of expert experience can substantially improve RL agent performance. We introduce Re:Frame (Retrieving Experience From Associative Memory), a plug-in module that augments a standard offline RL policy (e.g., Decision Transformer) with a small external Associative Memory Buffer (AMB) populated by expert trajectories drawn from a separate dataset. During training on low-quality data, the policy learns to retrieve expert data from the Associative Memory Buffer (AMB) via content-based associations and integrate them into decision-making; the same AMB is queried at evaluation. This requires no environment interaction and no modifications to the backbone architecture. On D4RL MuJoCo tasks, using as few as 60 expert trajectories (0.1% of a 6000-trajectory dataset), Re:Frame consistently improves over a strong Decision Transformer baseline in three of four settings, with gains up to +10.7 normalized points. These results show that Re:Frame offers a simple and data-efficient way to inject scarce expert knowledge and substantially improve offline RL from low-quality datasets. Show Chinese abstract

Abstract: 离线强化学习（RL）在收集大型专家数据集不可行或不实际时，通常会处理次优数据。 这种限制使得智能体难以泛化并实现高性能，因为它们必须主要从不完美或不一致的轨迹中学习。 因此，一个核心挑战是如何最好地利用稀缺的专家演示与大量但质量较低的数据。 我们证明，即使引入极少量的专家经验，也可以显著提高RL智能体的性能。 我们引入了Re:Frame（从关联记忆中检索经验），这是一个插件模块，通过一个由独立数据集中抽取的专家轨迹填充的小型外部关联记忆缓冲区（AMB），来增强标准的离线RL策略（例如，Decision Transformer）。 在低质量数据上训练时，策略通过基于内容的关联从关联记忆缓冲区（AMB）中检索专家数据，并将其整合到决策过程中；在评估时也查询相同的AMB。 这不需要环境交互，也不需要对主干架构进行修改。 在D4RL MuJoCo任务中，使用最少60个专家轨迹（一个6000轨迹数据集的0.1%），Re:Frame在四个设置中的三个中始终优于强大的Decision Transformer基线，在三个设置中提升高达+10.7归一化分数。 这些结果表明，Re:Frame提供了一种简单且数据高效的方法，可以注入稀缺的专家知识，并显著提高从低质量数据集中的离线RL性能。