Title: MimicDroid: In-Context Learning for Humanoid Robot Manipulation from Human Play Videos Show Chinese title

Title: MimicDroid：从人类游戏视频中进行上下文学习的人形机器人操作

Abstract: We aim to enable humanoid robots to efficiently solve new manipulation tasks from a few video examples. In-context learning (ICL) is a promising framework for achieving this goal due to its test-time data efficiency and rapid adaptability. However, current ICL methods rely on labor-intensive teleoperated data for training, which restricts scalability. We propose using human play videos -- continuous, unlabeled videos of people interacting freely with their environment -- as a scalable and diverse training data source. We introduce MimicDroid, which enables humanoids to perform ICL using human play videos as the only training data. MimicDroid extracts trajectory pairs with similar manipulation behaviors and trains the policy to predict the actions of one trajectory conditioned on the other. Through this process, the model acquired ICL capabilities for adapting to novel objects and environments at test time. To bridge the embodiment gap, MimicDroid first retargets human wrist poses estimated from RGB videos to the humanoid, leveraging kinematic similarity. It also applies random patch masking during training to reduce overfitting to human-specific cues and improve robustness to visual differences. To evaluate few-shot learning for humanoids, we introduce an open-source simulation benchmark with increasing levels of generalization difficulty. MimicDroid outperformed state-of-the-art methods and achieved nearly twofold higher success rates in the real world. Additional materials can be found on: ut-austin-rpl.github.io/MimicDroid Show Chinese abstract

Abstract: 我们旨在使人形机器人能够从少量视频示例中高效解决新的操作任务。 上下文学习（ICL）是一种有前景的框架，由于其测试时的数据效率和快速适应性，可以实现这一目标。 然而，当前的ICL方法依赖于劳动密集型的远程操作数据进行训练，这限制了可扩展性。 我们提出使用人类游戏视频——连续的、未标记的视频，展示人们自由地与环境互动——作为可扩展且多样的训练数据源。 我们引入了MimicDroid，它使人体机器人仅使用人类游戏视频作为训练数据即可执行ICL。 MimicDroid提取具有相似操作行为的轨迹对，并训练策略以另一个轨迹为条件预测其中一个轨迹的动作。 通过这个过程，模型在测试时获得了适应新物体和环境的ICL能力。 为了弥合具身差距，MimicDroid首先将从RGB视频中估计的人类手腕姿态重新定位到人形机器人，利用运动学相似性。 它还在训练期间应用随机补丁遮蔽，以减少对人类特定线索的过拟合并提高对视觉差异的鲁棒性。 为了评估人形机器人的少样本学习，我们引入了一个开源的模拟基准，具有逐步增加的泛化难度水平。 MimicDroid优于最先进的方法，在现实世界中实现了几乎两倍的成功率。 更多资料可在以下网址找到：ut-austin-rpl.github.io/MimicDroid