标题： 损失景观和超参数化非线性系统及神经网络中的优化 显示英文标题

标题： Loss landscapes and optimization in over-parameterized non-linear systems and neural networks

摘要： 深度学习的成功在很大程度上归功于应用于大型神经网络的基于梯度的优化方法的显著有效性。 本工作的目的是为过参数化的机器学习模型和非线性方程组中的损失景观和高效优化提出一种现代观点和一般的数学框架，这一设置包括过参数化的深度神经网络。 我们的初始观察是，这些系统的优化问题通常不是凸的，甚至在局部也不是。 我们认为它们满足PL$^*$，这是对大部分（但不是全部）参数空间上的Polyak-Lojasiewicz条件的一种变体，它保证了解的存在性和通过（随机）梯度下降（SGD/GD）的高效优化。 这些系统的PL$^*$条件与关联到非线性系统的切线核的条件数密切相关，这说明了基于PL$^*$的非线性理论如何平行于过参数化线性方程的经典分析。 我们证明宽神经网络满足PL$^*$条件，这解释了(S)GD收敛到全局最小值。 最后，我们提出了一个适用于“几乎”过参数化系统的PL$^*$条件的放松版本。 显示英文摘要

摘要： The success of deep learning is due, to a large extent, to the remarkable effectiveness of gradient-based optimization methods applied to large neural networks. The purpose of this work is to propose a modern view and a general mathematical framework for loss landscapes and efficient optimization in over-parameterized machine learning models and systems of non-linear equations, a setting that includes over-parameterized deep neural networks. Our starting observation is that optimization problems corresponding to such systems are generally not convex, even locally. We argue that instead they satisfy PL$^*$, a variant of the Polyak-Lojasiewicz condition on most (but not all) of the parameter space, which guarantees both the existence of solutions and efficient optimization by (stochastic) gradient descent (SGD/GD). The PL$^*$ condition of these systems is closely related to the condition number of the tangent kernel associated to a non-linear system showing how a PL$^*$-based non-linear theory parallels classical analyses of over-parameterized linear equations. We show that wide neural networks satisfy the PL$^*$ condition, which explains the (S)GD convergence to a global minimum. Finally we propose a relaxation of the PL$^*$ condition applicable to "almost" over-parameterized systems.