标题： 理解随机梯度Adam在学习神经网络中的泛化能力 显示英文标题

标题： Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks

摘要： Adam是深度学习中一种流行且广泛使用的自适应梯度方法，在理论研究中也受到了极大的关注。然而，大多数现有的理论工作主要分析其全批量版本，这与实际中使用的随机变体存在根本性差异。与SGD不同，即使在极小的学习率下，随机Adam也不会收敛到其全批量对应项。我们首次对批次大小如何影响Adam的泛化进行了理论分析，在图像数据上的两层过参数化卷积神经网络上进行了分析。我们的结果表明，尽管Adam和带有适当权重衰减的AdamW $\lambda$会收敛到较差的测试误差解，但它们的小批量变体可以达到接近零的测试误差。我们进一步证明，Adam的有效权重衰减边界严格小于AdamW，从理论上解释了为什么Adam需要更敏感的 $\lambda$ 调整。大量实验验证了我们的发现，展示了批次大小和权重衰减在Adam泛化性能中的关键作用。 显示英文摘要

摘要： Adam is a popular and widely used adaptive gradient method in deep learning, which has also received tremendous focus in theoretical research. However, most existing theoretical work primarily analyzes its full-batch version, which differs fundamentally from the stochastic variant used in practice. Unlike SGD, stochastic Adam does not converge to its full-batch counterpart even with infinitesimal learning rates. We present the first theoretical characterization of how batch size affects Adam's generalization, analyzing two-layer over-parameterized CNNs on image data. Our results reveal that while both Adam and AdamW with proper weight decay $\lambda$ converge to poor test error solutions, their mini-batch variants can achieve near-zero test error. We further prove Adam has a strictly smaller effective weight decay bound than AdamW, theoretically explaining why Adam requires more sensitive $\lambda$ tuning. Extensive experiments validate our findings, demonstrating the critical role of batch size and weight decay in Adam's generalization performance.