标题： 通过检索任务概念实现可证明的上下文向量算术 显示英文标题

标题： Provable In-Context Vector Arithmetic via Retrieving Task Concepts

摘要： 在上下文学习（ICL）中，其从演示中掌握功能/任务的能力引起了广泛关注。 最近的研究表明，在ICL期间，大型语言模型（LLM）中存在一个潜在的任务/功能向量。 Merullo等（2024）表明，LLM利用这个向量以及残差流进行类似Word2Vec的向量运算，解决事实回忆的ICL任务。 此外，近期的研究实证强调了问答数据在增强事实回忆能力中的关键作用。 尽管有这些见解，但理论解释仍然难以捉摸。 为了进一步推进，我们提出一个基于实证基础的层次概念建模的理论框架。 我们开发了一种优化理论，展示了通过交叉熵损失梯度下降训练的非线性残差变换器如何通过向量运算执行事实回忆的ICL任务。 我们证明了0-1损失收敛，并展示了强大的泛化能力，包括对概念重组和分布变化的鲁棒性。 这些结果阐明了变换器相对于静态嵌入前驱的优势。 实证模拟验证了我们的理论见解。 显示英文摘要

摘要： In-context learning (ICL) has garnered significant attention for its ability to grasp functions/tasks from demonstrations. Recent studies suggest the presence of a latent task/function vector in LLMs during ICL. Merullo et al. (2024) showed that LLMs leverage this vector alongside the residual stream for Word2Vec-like vector arithmetic, solving factual-recall ICL tasks. Additionally, recent work empirically highlighted the key role of Question-Answer data in enhancing factual-recall capabilities. Despite these insights, a theoretical explanation remains elusive. To move one step forward, we propose a theoretical framework building on empirically grounded hierarchical concept modeling. We develop an optimization theory, showing how nonlinear residual transformers trained via gradient descent on cross-entropy loss perform factual-recall ICL tasks via vector arithmetic. We prove 0-1 loss convergence and show the strong generalization, including robustness to concept recombination and distribution shifts. These results elucidate the advantages of transformers over static embedding predecessors. Empirical simulations corroborate our theoretical insights.