标题： 分层变分模仿学习控制程序 显示英文标题

标题： Hierarchical Variational Imitation Learning of Control Programs

摘要： 自主代理可以通过模仿教师演示的预期行为来学习。 分层控制策略在这种学习中普遍存在且有用，它有望将结构化任务分解为更简单的子任务，从而提高数据效率和泛化能力。 本文提出了一种变分推理方法，用于学习由参数化分层过程（PHP）表示的控制策略的模仿学习，PHP是一种类似程序的结构，在该结构中，过程可以调用子过程以执行子任务。 我们的方法通过学习过程调用和终止的潜在序列的近似后验分布，在教师演示的观察-动作轨迹的数据集中发现分层结构。 从这个学习到的分布中采样的样本随后指导分层控制策略的训练。 我们在分层模仿学习的背景下确定并展示了变分推理的一个新颖优势：在将策略分解为更简单的过程时，推理可以利用其他方法未使用的非因果信息。 使用变分推理训练PHP在数据效率和泛化方面优于LSTM基准模型，在执行冒泡排序算法时，所需数据不到一半即可达到24%的错误率，并且在执行Karel程序时实现零错误。 显示英文摘要

摘要： Autonomous agents can learn by imitating teacher demonstrations of the intended behavior. Hierarchical control policies are ubiquitously useful for such learning, having the potential to break down structured tasks into simpler sub-tasks, thereby improving data efficiency and generalization. In this paper, we propose a variational inference method for imitation learning of a control policy represented by parametrized hierarchical procedures (PHP), a program-like structure in which procedures can invoke sub-procedures to perform sub-tasks. Our method discovers the hierarchical structure in a dataset of observation-action traces of teacher demonstrations, by learning an approximate posterior distribution over the latent sequence of procedure calls and terminations. Samples from this learned distribution then guide the training of the hierarchical control policy. We identify and demonstrate a novel benefit of variational inference in the context of hierarchical imitation learning: in decomposing the policy into simpler procedures, inference can leverage acausal information that is unused by other methods. Training PHP with variational inference outperforms LSTM baselines in terms of data efficiency and generalization, requiring less than half as much data to achieve a 24% error rate in executing the bubble sort algorithm, and to achieve no error in executing Karel programs.