标题： 通过离策略发散正则化实现稳定的策略优化 显示英文标题

标题： Stable Policy Optimization via Off-Policy Divergence Regularization

摘要： 信任区域策略优化（TRPO）和近端策略优化（PPO）是深度强化学习（RL）中最为成功的策略梯度方法之一。虽然这些方法在广泛的任务中实现了最先进的性能，但在策略学习的稳定性以及如何使用离策略数据方面仍有改进空间。在本文中，我们重新审视这些算法的理论基础，并提出一种新算法，该算法通过一个邻近项稳定策略改进，该邻近项约束由连续策略产生的折扣状态-动作访问分布彼此接近。这个邻近项以访问分布之间的差异来表示，并以离策略和对抗的方式进行学习。我们通过实验表明，所提出的方法可以在基准高维控制任务中对稳定性产生有益影响并提高最终性能。 显示英文摘要

摘要： Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) are among the most successful policy gradient approaches in deep reinforcement learning (RL). While these methods achieve state-of-the-art performance across a wide range of challenging tasks, there is room for improvement in the stabilization of the policy learning and how the off-policy data are used. In this paper we revisit the theoretical foundations of these algorithms and propose a new algorithm which stabilizes the policy improvement through a proximity term that constrains the discounted state-action visitation distribution induced by consecutive policies to be close to one another. This proximity term, expressed in terms of the divergence between the visitation distributions, is learned in an off-policy and adversarial manner. We empirically show that our proposed method can have a beneficial effect on stability and improve final performance in benchmark high-dimensional control tasks.