标题： 向非线性方程组过度参数化系统的优化理论：深度学习的启示 显示英文标题

标题： Toward a theory of optimization for over-parameterized systems of non-linear equations: the lessons of deep learning

摘要： 深度学习的成功在很大程度上归功于在大型神经网络上应用的基于梯度的优化方法的显著效果。 在本工作中，我们隔离了一些一般的数学结构，使得在超参数化非线性方程系统中能够进行高效的优化，这一设置包括深度神经网络。 特别是，我们证明了对应于这些系统的优化问题即使在局部也是非凸的，但满足Polyak-Lojasiewicz（PL）条件，从而可以通过梯度下降或SGD进行高效优化。 我们将这些系统的PL条件与与切线核相关的条件数联系起来，并发展了一种非线性理论，与超参数化线性方程的经典分析平行。 我们讨论了这些思想如何应用于训练浅层和深层神经网络。 最后，我们指出，某些大系统的切线核可能即使在局部也远非常数。 然而，我们的分析仍然允许证明解的存在性和梯度下降和SGD的收敛性。 显示英文摘要

摘要： The success of deep learning is due, to a great extent, to the remarkable effectiveness of gradient-based optimization methods applied to large neural networks. In this work we isolate some general mathematical structures allowing for efficient optimization in over-parameterized systems of non-linear equations, a setting that includes deep neural networks. In particular, we show that optimization problems corresponding to such systems are not convex, even locally, but instead satisfy the Polyak-Lojasiewicz (PL) condition allowing for efficient optimization by gradient descent or SGD. We connect the PL condition of these systems to the condition number associated to the tangent kernel and develop a non-linear theory parallel to classical analyses of over-parameterized linear equations. We discuss how these ideas apply to training shallow and deep neural networks. Finally, we point out that tangent kernels associated to certain large system may be far from constant, even locally. Yet, our analysis still allows to demonstrate existence of solutions and convergence of gradient descent and SGD.