Title: Linear Frequency Principle Model to Understand the Absence of Overfitting in Neural Networks Show Chinese title

Title: 线性频率原则模型以理解神经网络中过拟合的缺失

Abstract: Why heavily parameterized neural networks (NNs) do not overfit the data is an important long standing open question. We propose a phenomenological model of the NN training to explain this non-overfitting puzzle. Our linear frequency principle (LFP) model accounts for a key dynamical feature of NNs: they learn low frequencies first, irrespective of microscopic details. Theory based on our LFP model shows that low frequency dominance of target functions is the key condition for the non-overfitting of NNs and is verified by experiments. Furthermore, through an ideal two-layer NN, we unravel how detailed microscopic NN training dynamics statistically gives rise to a LFP model with quantitative prediction power. Show Chinese abstract

Abstract: 为什么参数高度丰富的神经网络（NNs）不会过拟合数据是一个重要的长期悬而未决的问题。我们提出了一种神经网络训练的表观模型来解释这一不过拟合的谜题。我们的线性频率原则（LFP）模型解释了神经网络的一个关键动态特征：它们首先学习低频，而不管微观细节如何。基于我们LFP模型的理论表明，目标函数的低频主导是神经网络不过拟合的关键条件，并通过实验得到了验证。此外，通过一个理想的两层NN，我们揭示了详细的微观NN训练动力学如何统计上导致一个具有定量预测能力的LFP模型。