Title: Inverse Reinforcement Learning Using Just Classification and a Few Regressions Show Chinese title

Title: 使用分类和少量回归的逆强化学习

Abstract: Inverse reinforcement learning (IRL) aims to explain observed behavior by uncovering an underlying reward. In the maximum-entropy or Gumbel-shocks-to-reward frameworks, this amounts to fitting a reward function and a soft value function that together satisfy the soft Bellman consistency condition and maximize the likelihood of observed actions. While this perspective has had enormous impact in imitation learning for robotics and understanding dynamic choices in economics, practical learning algorithms often involve delicate inner-loop optimization, repeated dynamic programming, or adversarial training, all of which complicate the use of modern, highly expressive function approximators like neural nets and boosting. We revisit softmax IRL and show that the population maximum-likelihood solution is characterized by a linear fixed-point equation involving the behavior policy. This observation reduces IRL to two off-the-shelf supervised learning problems: probabilistic classification to estimate the behavior policy, and iterative regression to solve the fixed point. The resulting method is simple and modular across function approximation classes and algorithms. We provide a precise characterization of the optimal solution, a generic oracle-based algorithm, finite-sample error bounds, and empirical results showing competitive or superior performance to MaxEnt IRL. Show Chinese abstract

Abstract: 逆强化学习（IRL）旨在通过揭示潜在奖励来解释观察到的行为。 在最大熵或奖励的Gumbel扰动框架中，这相当于拟合一个奖励函数和一个软值函数，它们共同满足软贝尔曼一致性条件并最大化观察到动作的可能性。 虽然这种观点在机器人模仿学习和经济学中理解动态选择方面产生了巨大影响，但实际的学习算法通常涉及精细的内循环优化、重复的动态规划或对抗训练，所有这些都会使现代高度表达的函数逼近器（如神经网络和提升方法）的使用变得复杂。 我们重新审视softmax IRL，并表明总体最大似然解由涉及行为策略的线性固定点方程表征。 这一观察将IRL简化为两个现成的监督学习问题：概率分类用于估计行为策略，迭代回归用于求解固定点。 该方法在函数逼近类和算法上简单且模块化。 我们提供了最优解的精确表征，一种通用的基于oracle的算法，有限样本误差界限，并提供了实证结果，显示其性能与MaxEnt IRL相比具有竞争力或更优。