标题： 梯度下降能模拟提示吗？ 显示英文标题

标题： Can Gradient Descent Simulate Prompting?

摘要： 将新信息纳入语言模型（LM）的两种主要方法是更改其提示或更改其参数，例如通过微调。 参数更新不会为模型更改产生长期存储成本。 然而，对于许多模型更新，提示要有效得多：提示模型可以从单个示例中稳健地进行泛化，并得出在标准微调下不会出现的逻辑推理。 模型能否被修改，使得微调能模拟提示？ 本文描述了一种元训练LM的方法，使梯度更新能模拟对新信息进行条件处理的效果。 我们的方法使用基于梯度的元学习工具，但使用LM自身的提示预测作为目标，消除了对真实标签的需求。 随后的梯度下降训练恢复了某些（有时全部）提示模型的性能——在“反转诅咒”任务上表现出改进，并在单次梯度更新后回答关于文本段落的问题。 这些结果表明，通过适当的初始化，梯度下降可以出人意料地具有表现力。 我们的结果为长上下文建模提供了新的途径，并对基于梯度的学习的泛化能力提供了见解。 显示英文摘要

摘要： There are two primary ways of incorporating new information into a language model (LM): changing its prompt or changing its parameters, e.g. via fine-tuning. Parameter updates incur no long-term storage cost for model changes. However, for many model updates, prompting is significantly more effective: prompted models can generalize robustly from single examples and draw logical inferences that do not occur under standard fine-tuning. Can models be modified so that fine-tuning does emulate prompting? This paper describes a method for meta-training LMs such that gradient updates emulate the effects of conditioning on new information. Our approach uses tools from gradient-based meta-learning but uses an LM's own prompted predictions as targets, eliminating the need for ground-truth labels. Subsequent gradient descent training recovers some (and occasionally all) of prompted model performance -- showing improvement on the ``reversal curse'' tasks, and answering questions about text passages after a single gradient update. These results suggest that, with appropriate initialization, gradient descent can be surprisingly expressive. Our results suggest new avenues for long-context modeling and offer insight into the generalization capabilities of gradient-based learning.