标题： ACTLLM：动作一致性调优的大语言模型 显示英文标题

标题： ACTLLM: Action Consistency Tuned Large Language Model

摘要： 本文介绍了ACTLLM（动作一致性调优的大语言模型），这是一种用于动态环境中机器人操作的新方法。 传统的基于视觉的系统往往难以学习在任务执行和空间推理方面都表现出色的视觉表示，从而限制了它们在动态环境中的适应能力。 ACTLLM通过利用语言构建结构化的场景描述符来解决这些挑战，通过灵活的语言指令为空间理解和任务执行提供统一的接口。 此外，我们引入了一种新的动作一致性约束，将视觉感知与相应动作对齐，从而增强可操作视觉表示的学习。 另外，我们将操作任务的马尔可夫决策过程重新表述为多轮视觉对话框架。 这种方法使长期任务执行的建模成为可能，并从任务执行的历史中获得增强的上下文相关性。 在我们的评估中，ACTLLM在各种场景中表现出色，证明了其在具有挑战性的基于视觉的机器人操作任务中的有效性。 显示英文摘要

摘要： This paper introduces ACTLLM (Action Consistency Tuned Large Language Model), a novel approach for robot manipulation in dynamic environments. Traditional vision-based systems often struggle to learn visual representations that excel in both task execution and spatial reasoning, thereby limiting their adaptability in dynamic environments. ACTLLM addresses these challenges by harnessing language to craft structured scene descriptors, providing a uniform interface for both spatial understanding and task performance through flexible language instructions. Moreover, we introduce a novel action consistency constraint that aligns visual perception with corresponding actions, thereby enhancing the learning of actionable visual representations. Additionally, we have reformulated the Markov decision process for manipulation tasks into a multi-turn visual dialogue framework. This approach enables the modeling of long-term task execution with enhanced contextual relevance derived from the history of task execution. During our evaluation, ACTLLM excels in diverse scenarios, proving its effectiveness on challenging vision-based robot manipulation tasks.