标题： 带有自适应预条件的SGD：统一分析和动量加速 显示英文标题

标题： SGD with Adaptive Preconditioning: Unified Analysis and Momentum Acceleration

摘要： 在本文中，我们重新审视带有AdaGrad类型预条件的随机梯度下降（SGD）。我们的贡献有两方面。首先，我们在各向异性或矩阵光滑性和噪声假设下，开发了自适应预条件SGD的统一收敛分析。这使我们能够恢复几种流行的自适应梯度方法（包括AdaGrad-Norm、AdaGrad和ASGO/One-sided Shampoo）的最先进收敛结果。此外，我们建立了两种最近提出的算法Scion和DASGO之间的基本联系，并为后者提供了第一个理论保证。其次，我们表明，像AdaGrad和DASGO这样的方法的收敛性可以通过Nesterov动量证明加速，超过已知的最佳速率。因此，我们得到了第一个理论依据，即AdaGrad类型算法可以同时从对角预条件和动量中受益，这可能为Adam的实际效率提供最终解释。 显示英文摘要

摘要： In this paper, we revisit stochastic gradient descent (SGD) with AdaGrad-type preconditioning. Our contributions are twofold. First, we develop a unified convergence analysis of SGD with adaptive preconditioning under anisotropic or matrix smoothness and noise assumptions. This allows us to recover state-of-the-art convergence results for several popular adaptive gradient methods, including AdaGrad-Norm, AdaGrad, and ASGO/One-sided Shampoo. In addition, we establish the fundamental connection between two recently proposed algorithms, Scion and DASGO, and provide the first theoretical guarantees for the latter. Second, we show that the convergence of methods like AdaGrad and DASGO can be provably accelerated beyond the best-known rates using Nesterov momentum. Consequently, we obtain the first theoretical justification that AdaGrad-type algorithms can simultaneously benefit from both diagonal preconditioning and momentum, which may provide an ultimate explanation for the practical efficiency of Adam.