标题： 高维中SGD的均质化：精确动力学和泛化特性 显示英文标题

标题： Homogenization of SGD in high-dimensions: Exact dynamics and generalization properties

摘要： 我们开发了一个随机微分方程，称为均化SGD，用于分析高维随机最小二乘问题上带有$\ell^2$-正则化的随机梯度下降（SGD）的动力学。 我们证明了均化SGD是SGD的高维等价形式——对于任何二次统计量（例如，具有二次损失的总体风险），当样本数量$n$和特征数量$d$多项式相关（$d^c < n < d^{1/c}$对于某些$c > 0$）时，SGD迭代下的统计量会收敛到均化SGD下的统计量。 通过分析均化SGD，我们提供了SGD泛化性能的精确非渐近高维表达式，该表达式以一个Volterra积分方程的解来表示。 进一步地，我们在SGD训练的情况下，给出了二次损失下极限过量风险的精确值。 该分析适用于满足一组预解条件的数据矩阵和目标向量，这些条件可以粗略地看作是数据样本侧奇异向量的弱（非定量）去局域化形式。 提供了几个激励性的应用，包括独立样本的样本协方差矩阵和非生成模型目标的随机特征。 显示英文摘要

摘要： We develop a stochastic differential equation, called homogenized SGD, for analyzing the dynamics of stochastic gradient descent (SGD) on a high-dimensional random least squares problem with $\ell^2$-regularization. We show that homogenized SGD is the high-dimensional equivalence of SGD -- for any quadratic statistic (e.g., population risk with quadratic loss), the statistic under the iterates of SGD converges to the statistic under homogenized SGD when the number of samples $n$ and number of features $d$ are polynomially related ($d^c < n < d^{1/c}$ for some $c > 0$). By analyzing homogenized SGD, we provide exact non-asymptotic high-dimensional expressions for the generalization performance of SGD in terms of a solution of a Volterra integral equation. Further we provide the exact value of the limiting excess risk in the case of quadratic losses when trained by SGD. The analysis is formulated for data matrices and target vectors that satisfy a family of resolvent conditions, which can roughly be viewed as a weak (non-quantitative) form of delocalization of sample-side singular vectors of the data. Several motivating applications are provided including sample covariance matrices with independent samples and random features with non-generative model targets.