标题： 广义非策略演员-评论家 显示英文标题

标题： Generalized Off-Policy Actor-Critic

摘要： 我们提出了一种新的目标，即反事实目标，在持续强化学习（RL）设置中统一了现有用于非策略策略梯度算法的目标。 与通常使用的探险目标相比，当部署目标策略时，探险目标可能会误导其性能表现，而我们的新目标能更好地预测这种性能。 我们证明了广义非策略策略梯度定理，以计算反事实目标的策略梯度，并使用一种强调方法从此策略梯度中获得无偏样本，从而得到广义非策略演员-评论家（Geoff-PAC）算法。 我们在Mujoco机器人模拟任务中展示了Geoff-PAC相对于现有算法的优势，这是强调算法在当前深度强化学习基准中的首次实证成功。 显示英文摘要

摘要： We propose a new objective, the counterfactual objective, unifying existing objectives for off-policy policy gradient algorithms in the continuing reinforcement learning (RL) setting. Compared to the commonly used excursion objective, which can be misleading about the performance of the target policy when deployed, our new objective better predicts such performance. We prove the Generalized Off-Policy Policy Gradient Theorem to compute the policy gradient of the counterfactual objective and use an emphatic approach to get an unbiased sample from this policy gradient, yielding the Generalized Off-Policy Actor-Critic (Geoff-PAC) algorithm. We demonstrate the merits of Geoff-PAC over existing algorithms in Mujoco robot simulation tasks, the first empirical success of emphatic algorithms in prevailing deep RL benchmarks.