标题： 加速梯度下降优化和正则化的预处理 显示英文标题

标题： Preconditioning for Accelerated Gradient Descent Optimization and Regularization

摘要： 加速训练算法，如自适应学习率和各种归一化方法，被广泛使用但尚未完全理解。 当引入正则化时，标准优化器如自适应学习率可能无法有效工作。 这提出了对替代正则化方法的需求，并引发了如何正确将正则化与预处理相结合的问题。 在本文中，我们利用预处理理论来解决这些挑战，如下所述：(1) 我们解释了使用AdaGrad、RMSProp和Adam进行预处理如何加速训练；(2) 我们探讨了正则化与预处理之间的相互作用，概述了选择正则化变量的不同选项，并特别讨论了如何为梯度正则化实现这一点；(3) 我们展示了归一化方法如何通过改善Hessian条件来加速训练，并讨论了这种观点如何导致新的预处理训练算法。 我们的研究结果提供了一个统一的数学框架，用于理解各种加速技术并推导适当的正则化方案。 显示英文摘要

摘要： Accelerated training algorithms, such as adaptive learning rates and various normalization methods, are widely used but not fully understood. When regularization is introduced, standard optimizers like adaptive learning rates may not perform effectively. This raises the need for alternative regularization approaches and the question of how to properly combine regularization with preconditioning. In this paper, we address these challenges using the theory of preconditioning as follows: (1) We explain how preconditioning with AdaGrad, RMSProp, and Adam accelerates training; (2) We explore the interaction between regularization and preconditioning, outlining different options for selecting the variables for regularization, and in particular we discuss how to implement that for the gradient regularization; and (3) We demonstrate how normalization methods accelerate training by improving Hessian conditioning, and discuss how this perspective can lead to new preconditioning training algorithms. Our findings offer a unified mathematical framework for understanding various acceleration techniques and deriving appropriate regularization schemes.