Title: LoHoVLA: A Unified Vision-Language-Action Model for Long-Horizon Embodied Tasks Show Chinese title

Title: LoHoVLA：一种用于长时程具身任务的统一视觉-语言-动作模型

Abstract: Real-world embodied agents face long-horizon tasks, characterized by high-level goals demanding multi-step solutions beyond single actions. Successfully navigating these requires both high-level task planning (i.e., decomposing goals into sub-tasks) and low-level motion control (i.e., generating precise robot actions). While existing vision language action (VLA) models and hierarchical architectures offer potential in embodied tasks, the former often falter in planning, and the latter can suffer from coordination issues, both hampering performance. We introduce a new unified VLA framework for long-horizon tasks, dubbed LoHoVLA, to overcome these limitations. LoHoVLA leverages a large pretrained vision language model (VLM) as the backbone to jointly generate language and action tokens for sub-task generation and robot action prediction, respectively. This shared representation promotes better generalization across tasks. Additionally, LoHoVLA embraces a hierarchical closed-loop control mechanism to mitigate errors originating from both high-level planning and low-level control. To train LoHoVLA, we introduce LoHoSet, a dataset built on the Ravens simulator, containing 20 long-horizon tasks, each with 1,000 expert demonstrations composed of visual observations, linguistic goals, sub-tasks, and robot actions. Experimental results show that LoHoVLA significantly surpasses both hierarchical and standard VLA approaches on long-horizon embodied tasks in the Ravens simulator. These findings underscore the promise of unified architectures for advancing generalizable embodied intelligence. Show Chinese abstract

Abstract: 现实世界中的具身代理面临长期范围的任务，这些任务的特点是需要多步解决方案的高级目标，而单个动作无法完成。 成功应对这些挑战需要高水平的任务规划（即将目标分解为子任务）和低水平的运动控制（即生成精确的机器人动作）。 虽然现有的视觉语言动作（VLA）模型和分层架构在具身任务中展现出潜力，但前者在规划方面往往表现不佳，后者则可能遭受协调问题，这两者都限制了性能。 我们引入了一种新的统一视觉语言动作框架，称为LoHoVLA，以克服这些局限性。 LoHoVLA利用一个大规模预训练的视觉语言模型（VLM）作为主干，分别生成用于子任务生成的语言和动作标记以及机器人动作预测。 这种共享表示促进了跨任务的更好泛化。 此外，LoHoVLA采用了一个分层闭环控制机制，以减轻来自高层规划和低层控制的错误。 为了训练LoHoVLA，我们引入了LoHoSet，这是一个基于Ravens模拟器的数据集，包含20个长期范围任务，每个任务包含1,000个专家演示，由视觉观察、语言目标、子任务和机器人动作组成。 实验结果显示，LoHoVLA在Ravens模拟器中的长期具身任务上显著优于分层和标准VLA方法。 这些发现强调了统一架构在推进可推广的具身智能方面的潜力。