标题： 探索带有体积增长变换的RL游戏的分层空间结构 显示英文标题

标题： Exploring the Stratified Space Structure of an RL Game with the Volume Growth Transform

摘要： 在本工作中，我们探讨了为玩特定强化学习（RL）游戏而训练的变压器模型的嵌入空间结构。 具体而言，我们研究了基于变压器的近端策略优化（PPO）模型如何在一个简单环境中嵌入视觉输入，其中智能体必须收集“硬币”同时避免由“探照灯”组成的动态障碍物。 通过将Robinson等人对大型语言模型（LLM）的体积增长变换研究适应到强化学习设置中，我们发现，我们视觉硬币收集游戏的标记嵌入空间也不是一个流形，而是更适合建模为分层空间，其中局部维度可以从一点到另一点变化。 我们进一步通过证明相当一般的体积增长曲线可以通过分层空间实现来加强Robinson的方法。 最后，我们进行了一项分析，表明当强化学习智能体行动时，其潜在表示在遵循固定子策略期间在低局部维度时期和高局部维度时期之间交替，其中智能体实现了子目标（例如，收集一个物体）或环境复杂性增加（例如，出现更多障碍物）。 因此，我们的工作表明，分层潜在空间中的维度分布可能为强化学习游戏提供一种新的几何复杂性指标。 显示英文摘要

摘要： In this work, we explore the structure of the embedding space of a transformer model trained for playing a particular reinforcement learning (RL) game. Specifically, we investigate how a transformer-based Proximal Policy Optimization (PPO) model embeds visual inputs in a simple environment where an agent must collect "coins" while avoiding dynamic obstacles consisting of "spotlights." By adapting Robinson et al.'s study of the volume growth transform for LLMs to the RL setting, we find that the token embedding space for our visual coin collecting game is also not a manifold, and is better modeled as a stratified space, where local dimension can vary from point to point. We further strengthen Robinson's method by proving that fairly general volume growth curves can be realized by stratified spaces. Finally, we carry out an analysis that suggests that as an RL agent acts, its latent representation alternates between periods of low local dimension, while following a fixed sub-strategy, and bursts of high local dimension, where the agent achieves a sub-goal (e.g., collecting an object) or where the environmental complexity increases (e.g., more obstacles appear). Consequently, our work suggests that the distribution of dimensions in a stratified latent space may provide a new geometric indicator of complexity for RL games.