标题： 大型语言模型需要三维感知表示监督来进行场景理解 显示英文标题

标题： MLLMs Need 3D-Aware Representation Supervision for Scene Understanding

摘要： 场景理解的最新进展利用多模态大型语言模型（MLLM）通过其强大的2D预训练来进行3D推理。然而，在MLLM预训练过程中缺乏显式的3D数据限制了其3D表示能力。本文中，我们通过评估多视图对应关系来研究MLLM的3D意识，并揭示了3D感知表示的质量与下游任务性能之间存在强烈的正相关性。受此启发，我们提出了3DRS框架，通过引入来自预训练3D基础模型的监督来增强MLLM的3D表示学习。我们的方法将MLLM的视觉特征与从3D模型中提取的丰富的3D知识对齐，从而有效提升场景理解能力。广泛的实验表明，在多个基准数据集和MLLM上——包括视觉定位、图像描述生成以及问答任务——我们的方法均表现出一致的性能提升。项目页面：https://visual-ai.github.io/3drs 显示英文摘要

摘要： Recent advances in scene understanding have leveraged multimodal large language models (MLLMs) for 3D reasoning by capitalizing on their strong 2D pretraining. However, the lack of explicit 3D data during MLLM pretraining limits 3D representation capability. In this paper, we investigate the 3D-awareness of MLLMs by evaluating multi-view correspondence and reveal a strong positive correlation between the quality of 3D-aware representation and downstream task performance. Motivated by this, we propose 3DRS, a framework that enhances MLLM 3D representation learning by introducing supervision from pretrained 3D foundation models. Our approach aligns MLLM visual features with rich 3D knowledge distilled from 3D models, effectively improving scene understanding. Extensive experiments across multiple benchmarks and MLLMs -- including visual grounding, captioning, and question answering -- demonstrate consistent performance gains. Project page: https://visual-ai.github.io/3drs