Title: Hessian Eigenvectors and Principal Component Analysis of Neural Network Weight Matrices Show Chinese title

Title: Hessian特征向量和神经网络权重矩阵的主成分分析

Abstract: This study delves into the intricate dynamics of trained deep neural networks and their relationships with network parameters. Trained networks predominantly continue training in a single direction, known as the drift mode. This drift mode can be explained by the quadratic potential model of the loss function, suggesting a slow exponential decay towards the potential minima. We unveil a correlation between Hessian eigenvectors and network weights. This relationship, hinging on the magnitude of eigenvalues, allows us to discern parameter directions within the network. Notably, the significance of these directions relies on two defining attributes: the curvature of their potential wells (indicated by the magnitude of Hessian eigenvalues) and their alignment with the weight vectors. Our exploration extends to the decomposition of weight matrices through singular value decomposition. This approach proves practical in identifying critical directions within the Hessian, considering both their magnitude and curvature. Furthermore, our examination showcases the applicability of principal component analysis in approximating the Hessian, with update parameters emerging as a superior choice over weights for this purpose. Remarkably, our findings unveil a similarity between the largest Hessian eigenvalues of individual layers and the entire network. Notably, higher eigenvalues are concentrated more in deeper layers. Leveraging these insights, we venture into addressing catastrophic forgetting, a challenge of neural networks when learning new tasks while retaining knowledge from previous ones. By applying our discoveries, we formulate an effective strategy to mitigate catastrophic forgetting, offering a possible solution that can be applied to networks of varying scales, including larger architectures. Show Chinese abstract

Abstract: 本研究深入探讨了训练好的深度神经网络的复杂动态及其与网络参数之间的关系。 训练好的网络主要沿一个方向继续训练，称为漂移模式。 这种漂移模式可以通过损失函数的二次势模型来解释，表明其缓慢地指数衰减至势能极小值。 我们揭示了海森矩阵特征向量与网络权重之间的相关性。 这种关系依赖于特征值的大小，使我们能够辨别网络中的参数方向。 值得注意的是，这些方向的重要性取决于两个定义属性：势阱的曲率（由海森矩阵特征值的大小表示）以及它们与权重向量的对齐程度。 我们的研究扩展到通过奇异值分解对权重矩阵进行分解。 这种方法在考虑其大小和曲率的情况下，有助于识别海森矩阵中的关键方向。 此外，我们的研究展示了主成分分析在近似海森矩阵中的适用性，其中更新参数比权重更适合作为此目的的选择。 值得注意的是，我们的发现揭示了个别层和整个网络的最大海森特征值之间的相似性。 值得注意的是，较高的特征值更多集中在深层。 利用这些见解，我们尝试解决灾难性遗忘问题，这是神经网络在学习新任务的同时保留之前知识时面临的挑战。 通过应用我们的发现，我们制定了一种有效缓解灾难性遗忘的策略，提供了一个可应用于不同规模网络的可能解决方案，包括更大的架构。