标题： alpha-SVRG在强凸性下的收敛性分析 显示英文标题

标题： Convergence Analysis of alpha-SVRG under Strong Convexity

摘要： 随机一阶方法用于经验风险最小化，使用基于抽样数据的梯度近似来代替精确梯度。 这种构造会将噪声引入学习动态中，可以通过方差减少技术进行校正。 文献中有越来越多的证据表明，在许多现代学习应用中，噪声对优化和泛化可能具有有益的影响。 为此，最近提出的方差减少技术 alpha-SVRG [Yin 等，2023] 允许对学习动态中的残余噪声水平进行细粒度控制，并且在现代深度学习场景中已被报告在实验中优于 SGD 和 SVRG。 通过专注于强凸环境，我们首先在固定学习率下为 alpha-SVRG 提供了一个统一的收敛速率表达式，当设置 alpha=0 或 alpha=1 时，该表达式分别简化为 SGD 或 SVRG 的收敛速率。 我们证明，在适当选择 alpha 的情况下，alpha-SVRG 相比于 SGD 和 SVRG 具有更快的收敛速率。 在线性回归上的仿真结果验证了我们的理论。 显示英文摘要

摘要： Stochastic first-order methods for empirical risk minimization employ gradient approximations based on sampled data in lieu of exact gradients. Such constructions introduce noise into the learning dynamics, which can be corrected through variance-reduction techniques. There is increasing evidence in the literature that in many modern learning applications noise can have a beneficial effect on optimization and generalization. To this end, the recently proposed variance-reduction technique, alpha-SVRG [Yin et al., 2023] allows for fine-grained control of the level of residual noise in the learning dynamics, and has been reported to empirically outperform both SGD and SVRG in modern deep learning scenarios. By focusing on strongly convex environments, we first provide a unified convergence rate expression for alpha-SVRG under fixed learning rate, which reduces to that of either SGD or SVRG by setting alpha=0 or alpha=1, respectively. We show that alpha-SVRG has faster convergence rate compared to SGD and SVRG under suitable choice of alpha. Simulation results on linear regression validate our theory.