标题： 基于软约束对手的鲁棒离策强化学习 显示英文标题

标题： Robust off-policy Reinforcement Learning via Soft Constrained Adversary

摘要： 近年来，针对输入观测的鲁棒强化学习（RL）方法引起了广泛关注，并迅速发展，因为RL存在潜在的脆弱性。尽管这些先进方法取得了合理的成功，但在考虑长期范围内的对抗时存在两个局限性。 首先，策略与其对应最优对抗者之间的相互依赖关系限制了无策略RL算法的发展；虽然获得最优对抗者应该依赖当前策略，但这限制了无策略RL的应用。 其次，这些方法通常假设基于$L_p$-范数的扰动，即使在环境中已知扰动分布的情况下也是如此。 我们在此从另一个角度引入对抗性RL：一个带有先验知识分布的f-散度约束问题。 由此，我们推导出两种典型的攻击及其对应的鲁棒学习框架。 鲁棒性的评估显示，我们提出的方法在样本高效的无策略RL中表现出色。 显示英文摘要

摘要： Recently, robust reinforcement learning (RL) methods against input observation have garnered significant attention and undergone rapid evolution due to RL's potential vulnerability. Although these advanced methods have achieved reasonable success, there have been two limitations when considering adversary in terms of long-term horizons. First, the mutual dependency between the policy and its corresponding optimal adversary limits the development of off-policy RL algorithms; although obtaining optimal adversary should depend on the current policy, this has restricted applications to off-policy RL. Second, these methods generally assume perturbations based only on the $L_p$-norm, even when prior knowledge of the perturbation distribution in the environment is available. We here introduce another perspective on adversarial RL: an f-divergence constrained problem with the prior knowledge distribution. From this, we derive two typical attacks and their corresponding robust learning frameworks. The evaluation of robustness is conducted and the results demonstrate that our proposed methods achieve excellent performance in sample-efficient off-policy RL.