标题： 泰坦：在测试时学习记忆 显示英文标题

标题： Titans: Learning to Memorize at Test Time

摘要： 在过去十多年的研究中，人们一直在努力探索如何有效利用循环模型和注意力机制。 虽然循环模型旨在将数据压缩到固定大小的内存（称为隐藏状态），但注意力机制允许关注整个上下文窗口，捕捉所有标记的直接依赖关系。 然而，这种更精确的依赖关系建模伴随着二次成本，限制了模型只能处理固定长度的上下文。 我们提出了一个新的神经长期记忆模块，该模块学习记住历史上下文，并在利用过去信息的同时帮助注意力关注当前上下文。 我们表明，这种神经记忆在保持快速推理的同时，具有快速并行训练的优势。 从记忆的角度来看，我们认为由于注意力机制具有有限的上下文但精确的依赖关系建模，它表现为短期记忆，而神经记忆由于其能够记住数据，表现为长期、更持久的记忆。 基于这两个模块，我们引入了一种新的架构家族，称为 Titans，并提出了三种变体来解决如何有效地将记忆融入这种架构的问题。 我们在语言建模、常识推理、基因组学和时间序列任务上的实验结果表明，Titans 比 Transformers 和最近的现代线性循环模型更有效。 它们还能在比基线更高的准确性下有效地扩展到大于 2M 的上下文窗口大小，在针尖觅 haystack 任务中表现出色。 显示英文摘要

摘要： Over more than a decade there has been an extensive research effort on how to effectively utilize recurrent models and attention. While recurrent models aim to compress the data into a fixed-size memory (called hidden state), attention allows attending to the entire context window, capturing the direct dependencies of all tokens. This more accurate modeling of dependencies, however, comes with a quadratic cost, limiting the model to a fixed-length context. We present a new neural long-term memory module that learns to memorize historical context and helps attention to attend to the current context while utilizing long past information. We show that this neural memory has the advantage of fast parallelizable training while maintaining a fast inference. From a memory perspective, we argue that attention due to its limited context but accurate dependency modeling performs as a short-term memory, while neural memory due to its ability to memorize the data, acts as a long-term, more persistent, memory. Based on these two modules, we introduce a new family of architectures, called Titans, and present three variants to address how one can effectively incorporate memory into this architecture. Our experimental results on language modeling, common-sense reasoning, genomics, and time series tasks show that Titans are more effective than Transformers and recent modern linear recurrent models. They further can effectively scale to larger than 2M context window size with higher accuracy in needle-in-haystack tasks compared to baselines.