标题： 随机逼近中的块坐标最优步长 显示英文标题

标题： Stochastic Approximation with Block Coordinate Optimal Stepsizes

摘要： 我们考虑具有块坐标步长的随机逼近，并提出自适应步长规则，旨在最小化从当前迭代到最优点的期望距离。 这些步长规则利用每个块坐标上搜索方向的二阶矩的在线估计。 流行的Adam算法可以被解释为这种估计的一种特定启发式方法。 通过利用一个简单的条件估计器，我们推导出一种新方法，其性能与Adam相当，但需要更少的内存和更少的超参数。 我们证明了这一类方法几乎必然收敛到最优点的小邻域，邻域的半径取决于二阶矩估计器的偏差和方差。 我们的分析基于一个简单的瞄准条件，该条件既不假设凸性也不假设光滑性，因此具有广泛的应用性。 显示英文摘要

摘要： We consider stochastic approximation with block-coordinate stepsizes and propose adaptive stepsize rules that aim to minimize the expected distance from the next iterate to an optimal point. These stepsize rules employ online estimates of the second moment of the search direction along each block coordinate. The popular Adam algorithm can be interpreted as a particular heuristic for such estimation. By leveraging a simple conditional estimator, we derive a new method that obtains comparable performance as Adam but requires less memory and fewer hyper-parameters. We prove that this family of methods converges almost surely to a small neighborhood of the optimal point, and the radius of the neighborhood depends on the bias and variance of the second-moment estimator. Our analysis relies on a simple aiming condition that assumes neither convexity nor smoothness, thus has broad applicability.