标题： 神经网络中特征学习的微观和集体特征 显示英文标题

标题： Microscopic and collective signatures of feature learning in neural networks

摘要： 特征提取——识别数据相关属性的能力——是深度学习成功的关键因素。 然而，现有的预测理论很难阐明其本质，以至于对于特征学习的定义尚未达成共识。 在这个方向上，一个有希望的提示来自对神经网络训练动力学中准普遍特性的先前现象学观察，这些特性由特征几何的简单性质所展示。 我们在具有标准参数化的单隐层神经网络中的贝叶斯学习统计力学框架内解决这个问题。 在比例极限下（当网络宽度和训练集大小都很大时）的解析计算可以量化特征学习的指纹，包括集体特征（与流形几何相关）和微观特征（与权重相关）。 特别是，(i) 特征空间中不同类别流形之间的距离是温度的非单调函数，我们将其解释为在梯度下降（GD）动力学下观察到的现象的平衡对应物，以及 (ii) 网络中的微观可学习参数相对于无限宽度极限发生数据相关的位移，并发展出相关性。 这些结果表明，在后验预测分布为具有平凡重缩放先验的高斯过程回归的区域中，非平凡的特征学习正在发挥作用。 显示英文摘要

摘要： Feature extraction - the ability to identify relevant properties of data - is a key factor underlying the success of deep learning. Yet, it has proved difficult to elucidate its nature within existing predictive theories, to the extent that there is no consensus on the very definition of feature learning. A promising hint in this direction comes from previous phenomenological observations of quasi-universal aspects in the training dynamics of neural networks, displayed by simple properties of feature geometry. We address this problem within a statistical-mechanics framework for Bayesian learning in one hidden layer neural networks with standard parameterization. Analytical computations in the proportional limit (when both the network width and the size of the training set are large) can quantify fingerprints of feature learning, both collective ones (related to manifold geometry) and microscopic ones (related to the weights). In particular, (i) the distance between different class manifolds in feature space is a nonmonotonic function of the temperature, which we interpret as the equilibrium counterpart of a phenomenon observed under gradient descent (GD) dynamics, and (ii) the microscopic learnable parameters in the network undergo a finite data-dependent displacement with respect to the infinite-width limit, and develop correlations. These results indicate that nontrivial feature learning is at play in a regime where the posterior predictive distribution is that of Gaussian process regression with a trivially rescaled prior.