标题： 你的批量大小是问题所在吗？ 重温语言建模中的Adam-SGD差距 显示英文标题

标题： Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling

摘要： Adam 在语言模型中表现明显优于随机梯度下降（SGD），对此现象已有多种解释。在这项工作中，我们通过一系列全面调整的 Transformer 语言模型训练运行重新审视了这一“优化器差距”。我们系统地研究了动量、梯度裁剪和批量大小对 SGD 和 Adam 之间差距的影响。我们的实证结果显示，如果正确调优，带有动量的 SGD 实际上可以在小批量设置下表现出与 Adam 类似的性能。我们重新审视了现有解释 Adam 优势的因素，包括重尾类不平衡、方向锐度和 Hessian 异质性，这些因素难以直接解释该现象。为了弥合理解上的差距，通过对我们的 Transformer 训练运行以及受文献启发的简单二次设置进行分析，我们基于随机微分方程模型提供了新的见解，揭示了批量大小在训练动力学中的作用。 显示英文摘要

摘要： Adam is known to perform significantly better than Stochastic Gradient Descent (SGD) in language models, a phenomenon for which a number of explanations have been proposed. In this work, we revisit this "optimizer gap" through a series of comprehensively tuned baseline training runs for language modeling with Transformers. We exhaustively study how momentum, gradient clipping, and batch size affect the gap between SGD and Adam. Our empirical findings show that SGD with momentum can actually perform similarly to Adam in small-batch settings, if tuned correctly. We revisit existing explanations for Adam's advantage, including heavy-tailed class imbalance, directional sharpness, and Hessian heterogeneity, which struggle to directly explain this phenomenon. Towards bridging this gap in our understanding, by analyzing our Transformer training runs and simple quadratic settings inspired by the literature, we provide new insights, driven by stochastic differential equation models, into the role of batch size on the training dynamics.