标题： 人工代际智能：强化学习中的文化积累 显示英文标题

标题： Artificial Generational Intelligence: Cultural Accumulation in Reinforcement Learning

摘要： 文化积累推动了跨越人类历史的能力的无限和多样化进步。 它通过结合个体探索与代际信息传递，构建了一个不断扩展的知识和技能体系。 尽管在人类中取得了广泛的成功，但人工学习代理积累文化的能力仍缺乏深入研究。 特别是，强化学习的方法通常只关注单个生命周期内的改进。 现有的代际算法未能捕捉到文化积累的无限和涌现特性，这种特性使个体能够在创新与模仿之间进行权衡。 在之前展示的强化学习代理能够进行社会学习的基础上，我们发现，将这种学习与独立学习相结合的训练设置会引发文化积累。 这些积累的代理在相同的累计经验下，表现优于仅针对单个生命周期训练的代理。 我们通过构建两个模型来探索这种积累，这两个模型基于两种不同的世代概念：事件性世代，其中积累通过上下文学习发生；训练时间世代，其中积累通过权重内学习发生。 上下文和权重内的文化积累可以分别解释为知识和技能的积累。 据我们所知，这项工作是首次提出能够在强化学习中实现涌现文化积累的通用模型，这为更开放的学习系统开辟了新途径，同时也为建模人类文化提供了新的机会。 显示英文摘要

摘要： Cultural accumulation drives the open-ended and diverse progress in capabilities spanning human history. It builds an expanding body of knowledge and skills by combining individual exploration with inter-generational information transmission. Despite its widespread success among humans, the capacity for artificial learning agents to accumulate culture remains under-explored. In particular, approaches to reinforcement learning typically strive for improvements over only a single lifetime. Generational algorithms that do exist fail to capture the open-ended, emergent nature of cultural accumulation, which allows individuals to trade-off innovation and imitation. Building on the previously demonstrated ability for reinforcement learning agents to perform social learning, we find that training setups which balance this with independent learning give rise to cultural accumulation. These accumulating agents outperform those trained for a single lifetime with the same cumulative experience. We explore this accumulation by constructing two models under two distinct notions of a generation: episodic generations, in which accumulation occurs via in-context learning and train-time generations, in which accumulation occurs via in-weights learning. In-context and in-weights cultural accumulation can be interpreted as analogous to knowledge and skill accumulation, respectively. To the best of our knowledge, this work is the first to present general models that achieve emergent cultural accumulation in reinforcement learning, opening up new avenues towards more open-ended learning systems, as well as presenting new opportunities for modelling human culture.