标题： 策略镜像下降在兼容函数逼近之外的收敛性 显示英文标题

标题： Convergence of Policy Mirror Descent Beyond Compatible Function Approximation

摘要： 现代策略优化方法大致遵循策略镜像下降（PMD）算法模板，目前已有很多理论收敛结果。然而，大多数方法要么针对表格型环境，要么只有在所优化的策略类满足强闭包条件时才能有效应用，而这种情况在大规模环境中通常不成立。在本工作中，我们为一般的策略类开发了一个理论框架，其中我们将闭包条件替换为一个严格较弱的变分梯度支配假设，并得到了收敛到最优策略的速率上界。我们的主要结果利用了一个关于由当前策略的占用测度诱导的局部范数的平滑性新概念，并将PMD视为非欧几里得空间中平滑非凸优化的一个特例。 显示英文摘要

摘要： Modern policy optimization methods roughly follow the policy mirror descent (PMD) algorithmic template, for which there are by now numerous theoretical convergence results. However, most of these either target tabular environments, or can be applied effectively only when the class of policies being optimized over satisfies strong closure conditions, which is typically not the case when working with parametric policy classes in large-scale environments. In this work, we develop a theoretical framework for PMD for general policy classes where we replace the closure conditions with a strictly weaker variational gradient dominance assumption, and obtain upper bounds on the rate of convergence to the best-in-class policy. Our main result leverages a novel notion of smoothness with respect to a local norm induced by the occupancy measure of the current policy, and casts PMD as a particular instance of smooth non-convex optimization in non-Euclidean space.