Title: Natural Gradient VI: Guarantees for Non-Conjugate Models Show Chinese title

Title: 自然梯度VI：非共轭模型的保证

Abstract: Stochastic Natural Gradient Variational Inference (NGVI) is a widely used method for approximating posterior distribution in probabilistic models. Despite its empirical success and foundational role in variational inference, its theoretical underpinnings remain limited, particularly in the case of non-conjugate likelihoods. While NGVI has been shown to be a special instance of Stochastic Mirror Descent, and recent work has provided convergence guarantees using relative smoothness and strong convexity for conjugate models, these results do not extend to the non-conjugate setting, where the variational loss becomes non-convex and harder to analyze. In this work, we focus on mean-field parameterization and advance the theoretical understanding of NGVI in three key directions. First, we derive sufficient conditions under which the variational loss satisfies relative smoothness with respect to a suitable mirror map. Second, leveraging this structure, we propose a modified NGVI algorithm incorporating non-Euclidean projections and prove its global non-asymptotic convergence to a stationary point. Finally, under additional structural assumptions about the likelihood, we uncover hidden convexity properties of the variational loss and establish fast global convergence of NGVI to a global optimum. These results provide new insights into the geometry and convergence behavior of NGVI in challenging inference settings. Show Chinese abstract

Abstract: 随机自然梯度变分推断（NGVI）是一种在概率模型中近似后验分布的常用方法。 尽管其在经验上取得了成功并在变分推断中扮演着基础角色，但其理论基础仍然有限，特别是在非共轭似然的情况下。 虽然NGVI已被证明是随机镜面下降的一种特殊情况，并且最近的工作利用相对光滑性和强凸性为共轭模型提供了收敛保证，但这些结果无法扩展到非共轭设置，在这种情况下，变分损失变得非凸且更难分析。 在本工作中，我们专注于均场参数化，并在三个关键方向上推进了NGVI的理论理解。 首先，我们推导了变分损失相对于合适的镜面映射满足相对光滑性的充分条件。 其次，利用这一结构，我们提出了一种改进的NGVI算法，结合了非欧几里得投影，并证明了其全局非渐近收敛到一个平稳点。 最后，在对似然的额外结构假设下，我们揭示了变分损失的隐藏凸性性质，并建立了NGVI到全局最优值的快速全局收敛性。 这些结果为NGVI在具有挑战性的推断设置中的几何结构和收敛行为提供了新的见解。