标题： SAB3R：三维重建中的语义增强主干网络 显示英文标题

标题： SAB3R: Semantic-Augmented Backbone in 3D Reconstruction

摘要： 我们引入了一项新任务，称为“Map and Locate”，它统一了开放词汇分割（基于自然语言查询检测和分割对象实例）和三维重建（从视觉输入估计场景三维结构）这两个传统上截然不同的目标。 具体来说， Map and Locate 涉及从未定位的视频生成点云，并根据开放词汇查询分割对象实例。这项任务是迈向现实世界具身人工智能应用的关键一步，同时提出了一个连接重建、识别和重组的实际任务。 为了解决这一任务，我们提出了一种简单但有效的基线模型，命名为 SAB3R。 我们的方法基于最近在三维计算机视觉领域取得突破的 MASt3R，并结合了一种轻量级蒸馏策略。 该方法将来自二维视觉主干网络（例如 CLIP 和 DINOv2）的密集逐像素语义特征转移到 MASt3R 上，以增强其能力。 无需引入任何辅助冻结网络，我们的模型可以在单次前向传播中生成逐像素语义特征并构建一致的点地图。 与单独部署 MASt3R 和 CLIP 相比，我们的统一模型 SAB3R 在 Map and Locate 基准测试中表现出色。 此外，我们在二维语义分割和三维任务上评估了 SAB3R，以全面验证其有效性。 显示英文摘要

摘要： We introduce a new task, Map and Locate, which unifies the traditionally distinct objectives of open-vocabulary segmentation - detecting and segmenting object instances based on natural language queries - and 3D reconstruction, the process of estimating a scene's 3D structure from visual inputs. Specifically, Map and Locate involves generating a point cloud from an unposed video and segmenting object instances based on open-vocabulary queries. This task serves as a critical step toward real-world embodied AI applications and introduces a practical task that bridges reconstruction, recognition and reorganization. To tackle this task, we introduce a simple yet effective baseline, which we denote as SAB3R. Our approach builds upon MASt3R, a recent breakthrough in 3D computer vision, and incorporates a lightweight distillation strategy. This method transfers dense, per-pixel semantic features from 2D vision backbones (eg, CLIP and DINOv2) to enhance MASt3R's capabilities. Without introducing any auxiliary frozen networks, our model generates per-pixel semantic features and constructs cohesive point maps in a single forward pass. Compared to separately deploying MASt3R and CLIP, our unified model, SAB3R, achieves superior performance on the Map and Locate benchmark. Furthermore, we evaluate SAB3R on both 2D semantic segmentation and 3D tasks to comprehensively validate its effectiveness.