标题： 探索政策梯度中的神话 显示英文标题

标题： Behind the Myth of Exploration in Policy Gradients

摘要： 为了使用策略梯度算法计算接近最优的策略，实践中通常在学习目标中包含内在探索项。 尽管这些项的有效性通常由探索环境的内在需求来证明，但我们提出了一种新的分析方法，从数值优化的角度进行分析。 在学习目标上引入了两个标准，在其随机梯度估计上引入了另外两个标准，并随后用于讨论优化后的策略质量。 该分析揭示了探索技术的两个独立影响。 首先，它们使得学习目标平滑化，并在保留全局最大值的同时消除局部最优解。 其次，它们修改了梯度估计，增加了随机参数更新最终提供最优策略的可能性。 我们通过基于熵奖励的探索策略实证说明了这些影响，指出了局限性并提出了未来工作的方向。 显示英文摘要

摘要： In order to compute near-optimal policies with policy-gradient algorithms, it is common in practice to include intrinsic exploration terms in the learning objective. Although the effectiveness of these terms is usually justified by an intrinsic need to explore environments, we propose a novel analysis with the lens of numerical optimization. Two criteria are introduced on the learning objective and two others on its stochastic gradient estimates, and are afterwards used to discuss the quality of the policy after optimization. The analysis sheds light on two separate effects of exploration techniques. First, they make it possible to smooth the learning objective and to eliminate local optima while preserving the global maximum. Second, they modify the gradient estimates, increasing the probability that the stochastic parameter updates eventually provide an optimal policy. We empirically illustrate these effects with exploration strategies based on entropy bonuses, identifying limitations and suggesting directions for future work.