标题： 方向偏差有助于随机梯度下降在核回归模型中推广 显示英文标题

标题： The Directional Bias Helps Stochastic Gradient Descent to Generalize in Kernel Regression Models

摘要： 我们研究了非参数统计中的随机梯度下降（SGD）算法：特别是核回归。已知在线性回归设定下SGD的方向偏差性质被推广到核回归。更具体地，我们证明了具有适中且退火步长的SGD沿着Gram矩阵最大特征值对应的特征向量方向收敛。此外，具有适中或小步长的梯度下降（GD）沿着对应于最小特征值的特征向量方向收敛。这些事实被称为方向偏差性质；它们可以解释为何通过SGD计算出的估计器可能具有潜在更小的泛化误差。我们的理论应用通过模拟研究和基于FashionMNIST数据集的案例研究进行了展示。 显示英文摘要

摘要： We study the Stochastic Gradient Descent (SGD) algorithm in nonparametric statistics: kernel regression in particular. The directional bias property of SGD, which is known in the linear regression setting, is generalized to the kernel regression. More specifically, we prove that SGD with moderate and annealing step-size converges along the direction of the eigenvector that corresponds to the largest eigenvalue of the Gram matrix. In addition, the Gradient Descent (GD) with a moderate or small step-size converges along the direction that corresponds to the smallest eigenvalue. These facts are referred to as the directional bias properties; they may interpret how an SGD-computed estimator has a potentially smaller generalization error than a GD-computed estimator. The application of our theory is demonstrated by simulation studies and a case study that is based on the FashionMNIST dataset.