标题： MARBLE：多模态空间推理与规划的硬基准 显示英文标题

标题： MARBLE: A Hard Benchmark for Multimodal Spatial Reasoning and Planning

摘要： 处理来自多种模态的信息并逐步推理仍然是推动人工智能发展的关键挑战。 然而，现有的推理基准主要关注纯文本推理，或者使用可以通过直接从非文本模态中检索信息来回答的多模态问题。 因此，多模态领域中的复杂推理仍然理解不足。 在这里，我们提出了MARBLE，这是一个具有挑战性的多模态推理基准，旨在严格检验多模态语言模型（MLLMs）在通过复杂多模态问题和环境进行仔细逐步推理方面的能力。 MARBLE由两个高度具有挑战性的任务组成，即M-Portal和M-Cube，这些任务要求在空间、视觉和物理约束下制定和理解多步骤计划。 我们发现当前的MLLMs在MARBLE上的表现较差——所有12个先进模型在M-Portal上的表现接近随机，在M-Cube上的准确率为0%。 只有在简化的子任务中，一些模型才能超过随机基线，这表明复杂推理对于现有的MLLMs来说仍然是一个挑战。 此外，我们表明感知仍然是一个瓶颈，其中MLLMs偶尔无法从视觉输入中提取信息。 通过揭示MLLMs的局限性，我们希望MARBLE能推动下一代模型的发展，这些模型能够跨多个多模态推理步骤进行推理和规划。 显示英文摘要

摘要： The ability to process information from multiple modalities and to reason through it step-by-step remains a critical challenge in advancing artificial intelligence. However, existing reasoning benchmarks focus on text-only reasoning, or employ multimodal questions that can be answered by directly retrieving information from a non-text modality. Thus, complex reasoning remains poorly understood in multimodal domains. Here, we present MARBLE, a challenging multimodal reasoning benchmark that is designed to scrutinize multimodal language models (MLLMs) in their ability to carefully reason step-by-step through complex multimodal problems and environments. MARBLE is composed of two highly challenging tasks, M-Portal and M-Cube, that require the crafting and understanding of multistep plans under spatial, visual, and physical constraints. We find that current MLLMs perform poorly on MARBLE -- all the 12 advanced models obtain near-random performance on M-Portal and 0% accuracy on M-Cube. Only in simplified subtasks some models outperform the random baseline, indicating that complex reasoning is still a challenge for existing MLLMs. Moreover, we show that perception remains a bottleneck, where MLLMs occasionally fail to extract information from the visual inputs. By shedding a light on the limitations of MLLMs, we hope that MARBLE will spur the development of the next generation of models with the ability to reason and plan across many, multimodal reasoning steps.