标题： 注意小权重 显示英文标题

标题： Pay Attention to Small Weights

摘要： 微调大型预训练神经网络被证明在内存和计算成本方面都是资源密集型的。 为了缓解这个问题，一种常见的方法是将训练限制在模型参数的一个子集上。 通过分析微调过程中梯度与权重之间的关系，我们观察到一个显著的模式：大的梯度通常与小幅度的权重相关联。 这种相关性在微调设置中比从头开始训练更为明显。 受这一观察的启发，我们提出了NANOADAM，它在微调过程中仅动态更新小幅度的权重，并提供了几个实际优势：首先，这种标准是无梯度的——参数子集可以在不进行梯度计算的情况下确定；其次，它保留了大尺度的权重，这些权重可能编码了预训练期间学到的关键特征，从而减少了灾难性遗忘的风险；第三，它允许使用更大的学习率，并在实验中始终导致更好的泛化性能。 我们在自然语言处理和视觉任务中都验证了这一点。 显示英文摘要

摘要： Finetuning large pretrained neural networks is known to be resource-intensive, both in terms of memory and computational cost. To mitigate this, a common approach is to restrict training to a subset of the model parameters. By analyzing the relationship between gradients and weights during finetuning, we observe a notable pattern: large gradients are often associated with small-magnitude weights. This correlation is more pronounced in finetuning settings than in training from scratch. Motivated by this observation, we propose NANOADAM, which dynamically updates only the small-magnitude weights during finetuning and offers several practical advantages: first, this criterion is gradient-free -- the parameter subset can be determined without gradient computation; second, it preserves large-magnitude weights, which are likely to encode critical features learned during pretraining, thereby reducing the risk of catastrophic forgetting; thirdly, it permits the use of larger learning rates and consistently leads to better generalization performance in experiments. We demonstrate this for both NLP and vision tasks.