标题： 离线策略 actor-critic 算法的双重稳健性研究 显示英文标题

标题： Doubly Robust Off-Policy Actor-Critic Algorithms for Reinforcement Learning

摘要： 我们研究了几个变体下的基于值的异策略演员-评论家算法中的异策略评论家评估问题。异策略演员-评论家算法需要一个异策略评论家评估步骤，以估计每次策略梯度更新后新策略的价值。尽管异策略策略梯度在控制任务上取得了巨大成功，但现有的通用方法存在高方差和不稳定的问题，部分原因是策略改进依赖于估计值函数的梯度。在这项工作中，我们提出了演员-评论家中一种新的异策略策略评估方法，基于双重鲁棒估计器。我们将双重鲁棒估计器从异策略策略评估（OPE）扩展到由奖励估计性能模型组成的演员-评论家算法。我们发现，评论家的双重鲁棒估计可以显著提高连续控制任务中的性能。此外，在奖励函数是随机的且可能导致高方差的情况下，双重鲁棒评论家估计可以在被污染的、随机的奖励信号下改善性能，表明它对于鲁棒和安全的强化学习具有实用性。 显示英文摘要

摘要： We study the problem of off-policy critic evaluation in several variants of value-based off-policy actor-critic algorithms. Off-policy actor-critic algorithms require an off-policy critic evaluation step, to estimate the value of the new policy after every policy gradient update. Despite enormous success of off-policy policy gradients on control tasks, existing general methods suffer from high variance and instability, partly because the policy improvement depends on gradient of the estimated value function. In this work, we present a new way of off-policy policy evaluation in actor-critic, based on the doubly robust estimators. We extend the doubly robust estimator from off-policy policy evaluation (OPE) to actor-critic algorithms that consist of a reward estimator performance model. We find that doubly robust estimation of the critic can significantly improve performance in continuous control tasks. Furthermore, in cases where the reward function is stochastic that can lead to high variance, doubly robust critic estimation can improve performance under corrupted, stochastic reward signals, indicating its usefulness for robust and safe reinforcement learning.