标题： ODICE：通过正交梯度更新揭示分布校正估计的奥秘 显示英文标题

标题： ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Update

摘要： 在本研究中，我们调查了分布校正估计（DICE）方法，这是离线强化学习（RL）和模仿学习（IL）中的一个重要研究方向。 基于DICE的方法施加状态-动作级别的行为约束，这是离线学习的理想选择。 然而，它们通常比仅使用动作级别行为约束的当前最先进（SOTA）方法表现差得多。 在重新审视基于DICE的方法后，我们发现当使用真实梯度更新学习价值函数时，存在两个梯度项：前向梯度（在当前状态上取）和后向梯度（在下一个状态上取）。 使用前向梯度与许多离线RL方法有很大相似性，因此可以视为应用动作级别约束。 然而，如果这两个梯度方向冲突，直接添加后向梯度可能会导致效果退化或抵消其影响。 为了解决这个问题，我们提出了一种简单而有效的修改，将后向梯度投影到前向梯度的法平面，从而得到一个正交梯度更新，这是基于DICE方法的新学习规则。 我们进行了详尽的理论分析，发现投影后的后向梯度带来了状态级别的行为正则化，这揭示了基于DICE方法的奥秘：价值学习目标确实试图施加状态-动作级别的约束，但需要以修正的方式使用。 通过简单的例子和在复杂离线RL和IL任务上的广泛实验，我们证明了使用正交梯度更新的基于DICE方法（O-DICE）达到了SOTA性能和出色的鲁棒性。 显示英文摘要

摘要： In this study, we investigate the DIstribution Correction Estimation (DICE) methods, an important line of work in offline reinforcement learning (RL) and imitation learning (IL). DICE-based methods impose state-action-level behavior constraint, which is an ideal choice for offline learning. However, they typically perform much worse than current state-of-the-art (SOTA) methods that solely use action-level behavior constraint. After revisiting DICE-based methods, we find there exist two gradient terms when learning the value function using true-gradient update: forward gradient (taken on the current state) and backward gradient (taken on the next state). Using forward gradient bears a large similarity to many offline RL methods, and thus can be regarded as applying action-level constraint. However, directly adding the backward gradient may degenerate or cancel out its effect if these two gradients have conflicting directions. To resolve this issue, we propose a simple yet effective modification that projects the backward gradient onto the normal plane of the forward gradient, resulting in an orthogonal-gradient update, a new learning rule for DICE-based methods. We conduct thorough theoretical analyses and find that the projected backward gradient brings state-level behavior regularization, which reveals the mystery of DICE-based methods: the value learning objective does try to impose state-action-level constraint, but needs to be used in a corrected way. Through toy examples and extensive experiments on complex offline RL and IL tasks, we demonstrate that DICE-based methods using orthogonal-gradient updates (O-DICE) achieve SOTA performance and great robustness.