标题： PARTNR：具身多智能体任务中规划与推理的基准 显示英文标题

标题： PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks

摘要： 我们提出了一个用于人机协作（humaN-Robot collaboration）的规划与推理任务基准测试集 PARTNR，旨在研究家庭活动中的机器人-人类协调。 PARTNR 任务展示了日常任务的特性，例如空间、时间以及异构代理能力的限制。 我们采用使用大型语言模型（LLMs）的半自动化任务生成管道，并结合模拟以实现接地和验证。 PARTNR 是同类中最大的基准测试集，包含 100,000 个自然语言任务，覆盖 60 个房屋和 5,819 个独特对象。 我们在 PARTNR 任务上分析了最先进的 LLMs 在规划、感知和技能执行方面的表现。 分析揭示了最先进的模型存在显著局限性，例如协作能力差以及在任务跟踪和从错误中恢复方面的问题。 当 LLMs 与真实人类配对时，它们需要比两个人类协作多 1.5 倍的步骤，比单个人类多 1.1 倍的步骤，这突显了这些模型改进的空间。 我们进一步表明，用规划数据微调较小的 LLMs 可以实现与大 9 倍的模型相当的性能，同时在推理速度上快 8.6 倍。 总体而言， PARTNR 强调了协作具身代理所面临的重大挑战，并旨在推动这一方向的研究。 显示英文摘要

摘要： We present a benchmark for Planning And Reasoning Tasks in humaN-Robot collaboration (PARTNR) designed to study human-robot coordination in household activities. PARTNR tasks exhibit characteristics of everyday tasks, such as spatial, temporal, and heterogeneous agent capability constraints. We employ a semi-automated task generation pipeline using Large Language Models (LLMs), incorporating simulation in the loop for grounding and verification. PARTNR stands as the largest benchmark of its kind, comprising 100,000 natural language tasks, spanning 60 houses and 5,819 unique objects. We analyze state-of-the-art LLMs on PARTNR tasks, across the axes of planning, perception and skill execution. The analysis reveals significant limitations in SoTA models, such as poor coordination and failures in task tracking and recovery from errors. When LLMs are paired with real humans, they require 1.5x as many steps as two humans collaborating and 1.1x more steps than a single human, underscoring the potential for improvement in these models. We further show that fine-tuning smaller LLMs with planning data can achieve performance on par with models 9 times larger, while being 8.6x faster at inference. Overall, PARTNR highlights significant challenges facing collaborative embodied agents and aims to drive research in this direction.