标题： PanSt3R：多视角一致的全景分割 显示英文标题

标题： PanSt3R: Multi-view Consistent Panoptic Segmentation

摘要： 3D场景的全景分割，涉及对场景密集3D重建中的物体实例进行分割和分类，是一个具有挑战性的问题，尤其是在仅依赖未定位的2D图像时。现有的方法通常利用现成的模型提取每帧的2D全景分割，然后优化一个隐式几何表示（通常基于NeRF）以整合和融合2D预测。我们认为，对于一个本质上是3D且多视角的问题，依赖2D全景分割可能是次优的，因为它未能充分利用跨视角的空间关系。除了需要相机参数外，这些方法还为每个场景需要计算昂贵的测试时优化。相反，在这项工作中，我们提出了一种统一且集成的方法PanSt3R，通过在一次前向传递中联合预测3D几何和多视角全景分割，消除了测试时优化的需要。我们的方法建立在最近的3D重建进展之上，特别是MUSt3R，这是DUSt3R的一个可扩展的多视角版本，并通过语义意识和多视角全景分割能力对其进行增强。我们还重新审视了标准的后处理掩码合并过程，并引入了一种更系统化的多视角分割方法。我们还介绍了一种基于PanSt3R和普通3DGS预测生成新视角预测的简单方法。总体而言，所提出的PanSt3R在概念上简单，但快速且可扩展，在多个基准测试中达到了最先进性能，同时比现有方法快几个数量级。 显示英文摘要

摘要： Panoptic segmentation of 3D scenes, involving the segmentation and classification of object instances in a dense 3D reconstruction of a scene, is a challenging problem, especially when relying solely on unposed 2D images. Existing approaches typically leverage off-the-shelf models to extract per-frame 2D panoptic segmentations, before optimizing an implicit geometric representation (often based on NeRF) to integrate and fuse the 2D predictions. We argue that relying on 2D panoptic segmentation for a problem inherently 3D and multi-view is likely suboptimal as it fails to leverage the full potential of spatial relationships across views. In addition to requiring camera parameters, these approaches also necessitate computationally expensive test-time optimization for each scene. Instead, in this work, we propose a unified and integrated approach PanSt3R, which eliminates the need for test-time optimization by jointly predicting 3D geometry and multi-view panoptic segmentation in a single forward pass. Our approach builds upon recent advances in 3D reconstruction, specifically upon MUSt3R, a scalable multi-view version of DUSt3R, and enhances it with semantic awareness and multi-view panoptic segmentation capabilities. We additionally revisit the standard post-processing mask merging procedure and introduce a more principled approach for multi-view segmentation. We also introduce a simple method for generating novel-view predictions based on the predictions of PanSt3R and vanilla 3DGS. Overall, the proposed PanSt3R is conceptually simple, yet fast and scalable, and achieves state-of-the-art performance on several benchmarks, while being orders of magnitude faster than existing methods.