Title: Evaluating Robot Policies in a World Model Show Chinese title

Title: 在世界模型中评估机器人策略

Abstract: Robotics has broad applications from automating house chores to taking care of patients. However, evaluating robot control policies is challenging, as real-world testing is expensive, while handcrafted simulations often fail to accurately reflect real-world conditions, resulting in poor correlation between simulated evaluation and real-world outcomes. In this work, we investigate World-model-based Policy Evaluation (WPE). We first train an action-conditioned video generation model as a proxy to real-world environments. To enable efficient rollouts of hundreds of interactive steps while mitigating error accumulation in the world model, we propose an inference scheme which we call Blockwise-Autoregressive Diffusion Transformer with adjustable context and decoding horizon lengths. To ensure that the world model indeed follows action input, we propose metrics based on the agreement between the ground truth video and generated video conditioned on the same sequence of actions to evaluate the world model. We then use the world model for policy evaluation by performing Monte Carlo rollouts in the world model while employing a vision-language model (VLM) as a reward function. Interestingly, we found that WPE tends to underestimate the policy values for in-distribution actions and overestimate policy values for out-of-distribution actions. Nevertheless, WPE preserves the relative rankings of different policies. In emulating real robot executions, WPE achieves high fidelity in mimicing robot arm movements as in real videos, while emulating highly realistic object interaction remains challenging. Despite this limitation, we show that a world model can serve as a starting point for evaluating robot policies before real-world deployment. Show Chinese abstract

Abstract: 机器人技术从自动化家庭杂务到照顾病人有着广泛的应用。 然而，评估机器人控制策略具有挑战性，因为现实世界的测试成本高昂，而手工设计的模拟往往无法准确反映真实世界的情况，导致模拟评估与实际结果之间相关性较差。 在这项工作中，我们研究了基于世界模型的策略评估（WPE）。 我们首先训练一个以动作为条件的视频生成模型，作为真实世界环境的代理。 为了能够在世界模型中高效地执行数百个交互步骤，并减轻误差累积，我们提出了一种推理方案，称为可调上下文和解码范围长度的块自回归扩散变换器。 为了确保世界模型确实遵循动作输入，我们提出了基于真实视频和相同动作序列生成视频之间一致性的指标来评估世界模型。 然后，我们在世界模型中使用蒙特卡罗滚动，并采用视觉语言模型（VLM）作为奖励函数来进行策略评估。 有趣的是，我们发现WPE倾向于低估分布内动作的策略值，而高估分布外动作的策略值。 尽管如此，WPE保留了不同策略之间的相对排名。 在模拟真实机器人执行时， WPE在模仿机器人手臂运动方面达到了很高的保真度，而在模仿高度逼真的物体交互方面仍然具有挑战性。 尽管存在这一限制，我们展示了世界模型可以作为评估机器人策略的起点，以便在真实部署之前进行评估。