标题： 多类随机梯度下降的精确动力学 显示英文标题

标题： Exact Dynamics of Multi-class Stochastic Gradient Descent

摘要： 我们开发了一个框架，用于分析使用单次遍历随机梯度下降（SGD）训练的多种高维优化问题的训练和学习率动态，这些数据来自多个各向异性类别。 我们给出了限制动态的一类函数的精确表达式，包括风险和与真实信号的重叠，这些表达式是基于一个常微分方程组（ODE）的确定性解。 我们将现有的高维SGD动态理论扩展到高斯混合数据和大量（随着参数规模增长）类别。 然后，我们详细研究了数据协方差的各向异性结构在二元逻辑回归和最小二乘损失问题中的影响。 我们研究了三种情况：各向同性协方差、具有大量零特征值的数据协方差矩阵（称为零一模型），以及谱遵循幂律分布的协方差矩阵。 我们证明存在一种结构相变。 特别是，我们证明对于零一模型和足够大的幂的幂律模型，SGD倾向于更紧密地对齐于投影到“干净方向”（即方差较小的方向）的类别均值。 这得到了数值模拟和分析研究的支持，这些研究显示了高维极限下损失的精确渐近行为。 显示英文摘要

摘要： We develop a framework for analyzing the training and learning rate dynamics on a variety of high- dimensional optimization problems trained using one-pass stochastic gradient descent (SGD) with data generated from multiple anisotropic classes. We give exact expressions for a large class of functions of the limiting dynamics, including the risk and the overlap with the true signal, in terms of a deterministic solution to a system of ODEs. We extend the existing theory of high-dimensional SGD dynamics to Gaussian-mixture data and a large (growing with the parameter size) number of classes. We then investigate in detail the effect of the anisotropic structure of the covariance of the data in the problems of binary logistic regression and least square loss. We study three cases: isotropic covariances, data covariance matrices with a large fraction of zero eigenvalues (denoted as the zero-one model), and covariance matrices with spectra following a power-law distribution. We show that there exists a structural phase transition. In particular, we demonstrate that, for the zero-one model and the power-law model with sufficiently large power, SGD tends to align more closely with values of the class mean that are projected onto the "clean directions" (i.e., directions of smaller variance). This is supported by both numerical simulations and analytical studies, which show the exact asymptotic behavior of the loss in the high-dimensional limit.