Title: Overparametrization bends the landscape: BBP transitions at initialization in simple Neural Networks Show Chinese title

Title: 过参数化扭曲了景观：简单神经网络初始化时的BBP转变

Abstract: High-dimensional non-convex loss landscapes play a central role in the theory of Machine Learning. Gaining insight into how these landscapes interact with gradient-based optimization methods, even in relatively simple models, can shed light on this enigmatic feature of neural networks. In this work, we will focus on a prototypical simple learning problem, which generalizes the Phase Retrieval inference problem by allowing the exploration of overparametrized settings. Using techniques from field theory, we analyze the spectrum of the Hessian at initialization and identify a Baik-Ben Arous-P\'ech\'e (BBP) transition in the amount of data that separates regimes where the initialization is informative or uninformative about a planted signal of a teacher-student setup. Crucially, we demonstrate how overparameterization can bend the loss landscape, shifting the transition point, even reaching the information-theoretic weak-recovery threshold in the large overparameterization limit, while also altering its qualitative nature. We distinguish between continuous and discontinuous BBP transitions and support our analytical predictions with simulations, examining how they compare to the finite-N behavior. In the case of discontinuous BBP transitions strong finite-N corrections allow the retrieval of information at a signal-to-noise ratio (SNR) smaller than the predicted BBP transition. In these cases we provide estimates for a new lower SNR threshold that marks the point at which initialization becomes entirely uninformative. Show Chinese abstract

Abstract: 高维非凸损失景观在机器学习理论中起着核心作用。 了解这些景观如何与基于梯度的优化方法相互作用，即使在相对简单的模型中，也可以揭示神经网络这一神秘特征。 在本工作中，我们将关注一个典型的简单学习问题，该问题通过允许探索过参数化设置来推广相位检索推理问题。 使用场论技术，我们分析了初始化时的Hessian谱，并确定了数据量的BBP转变，该转变区分了初始化对教师-学生设置中植入信号有信息或无信息的区域。 关键的是，我们展示了过参数化如何弯曲损失景观，改变转变点，甚至在大过参数化极限下达到信息理论上的弱恢复阈值，同时改变其定性性质。 我们区分了连续和不连续的BBP转变，并通过模拟支持我们的分析预测，研究它们与有限N行为的比较。 在不连续BBP转变的情况下，强烈的有限N修正允许在信噪比（SNR）小于预测的BBP转变时恢复信息。 在这些情况下，我们提供了新的更低SNR阈值的估计，该阈值标记了初始化完全无信息的点。