标题： $ε$-阶和阶梯现象：对神经网络训练动态的新见解 显示英文标题

标题： $ε$-rank and the Staircase Phenomenon: New Insights into Neural Network Training Dynamics

摘要： 理解深度神经网络 (DNN) 的训练动态，尤其是它们如何从高维数据演化出低维特征，仍然是深度学习理论的核心挑战。 本文引入了 $\epsilon$-rank 的概念，这是一个量化终端隐藏层神经元函数有效特征的全新指标。 通过对不同任务的广泛实验，我们观察到一种普遍存在的阶梯现象：在使用标准随机梯度下降方法进行训练的过程中，损失函数的下降伴随着 $\epsilon$-rank 的上升，并呈现出阶梯状模式。 从理论上，我们严格证明了损失下限与 $\epsilon$-rank 之间存在负相关性，表明较高的 $\epsilon$-rank 对于显著降低损失至关重要。 此外，数值证据表明，在同一个深度神经网络中，后续隐藏层的 $\epsilon$-rank 高于前一个隐藏层。 基于这些观察，为了消除阶梯现象，我们提出了一种新颖的初始隐藏层预训练策略，以提高终端隐藏层的$\epsilon$秩。 数值实验验证了该策略在缩短训练时间并提高各种任务的准确率方面的有效性。 因此，新引入的$\epsilon$秩概念是一个可计算的量，可作为深度神经网络的内在有效度量特征，为理解神经网络的训练动态提供了一个新的视角，并为在实际应用中设计高效的训练策略提供了理论基础。 显示英文摘要

摘要： Understanding the training dynamics of deep neural networks (DNNs), particularly how they evolve low-dimensional features from high-dimensional data, remains a central challenge in deep learning theory. In this work, we introduce the concept of $\epsilon$-rank, a novel metric quantifying the effective feature of neuron functions in the terminal hidden layer. Through extensive experiments across diverse tasks, we observe a universal staircase phenomenon: during training process implemented by the standard stochastic gradient descent methods, the decline of the loss function is accompanied by an increase in the $\epsilon$-rank and exhibits a staircase pattern. Theoretically, we rigorously prove a negative correlation between the loss lower bound and $\epsilon$-rank, demonstrating that a high $\epsilon$-rank is essential for significant loss reduction. Moreover, numerical evidences show that within the same deep neural network, the $\epsilon$-rank of the subsequent hidden layer is higher than that of the previous hidden layer. Based on these observations, to eliminate the staircase phenomenon, we propose a novel pre-training strategy on the initial hidden layer that elevates the $\epsilon$-rank of the terminal hidden layer. Numerical experiments validate its effectiveness in reducing training time and improving accuracy across various tasks. Therefore, the newly introduced concept of $\epsilon$-rank is a computable quantity that serves as an intrinsic effective metric characteristic for deep neural networks, providing a novel perspective for understanding the training dynamics of neural networks and offering a theoretical foundation for designing efficient training strategies in practical applications.