标题： SurgTPGS：具有文本提示的高斯点云语义3D外科场景理解 显示英文标题

标题： SurgTPGS: Semantic 3D Surgical Scene Understanding with Text Promptable Gaussian Splatting

摘要： 在当代外科研究和实践中，具有文本提示能力的准确理解三维外科场景对于外科规划和实时术中导航尤其重要，在这里精确识别和交互外科工具和解剖结构是至关重要的。然而，现有工作分别关注外科视觉-语言模型（VLM）、三维重建和分割，缺乏对实时文本提示三维查询的支持。在本文中，我们提出了SurgTPGS，一种新颖的文本提示高斯点云方法来填补这一空白。我们引入了一种结合Segment Anything模型和最先进的视觉-语言模型的三维语义特征学习策略。我们提取了用于三维外科场景重建的分割语言特征，使对复杂外科环境的理解更加深入。我们还提出了语义感知变形跟踪，以捕捉语义特征的无缝变形，为纹理和语义特征提供更精确的重建。此外，我们提出了语义区域感知优化，利用基于区域的语义信息来监督训练，特别促进了重建质量和语义平滑性。我们在两个真实世界的外科数据集上进行了全面实验，以证明SurgTPGS优于最先进方法，突显了其革新外科实践的潜力。SurgTPGS通过提高外科精度和安全性，为开发下一代智能外科系统铺平了道路。我们的代码可在以下地址获取：https://github.com/lastbasket/SurgTPGS. 显示英文摘要

摘要： In contemporary surgical research and practice, accurately comprehending 3D surgical scenes with text-promptable capabilities is particularly crucial for surgical planning and real-time intra-operative guidance, where precisely identifying and interacting with surgical tools and anatomical structures is paramount. However, existing works focus on surgical vision-language model (VLM), 3D reconstruction, and segmentation separately, lacking support for real-time text-promptable 3D queries. In this paper, we present SurgTPGS, a novel text-promptable Gaussian Splatting method to fill this gap. We introduce a 3D semantics feature learning strategy incorporating the Segment Anything model and state-of-the-art vision-language models. We extract the segmented language features for 3D surgical scene reconstruction, enabling a more in-depth understanding of the complex surgical environment. We also propose semantic-aware deformation tracking to capture the seamless deformation of semantic features, providing a more precise reconstruction for both texture and semantic features. Furthermore, we present semantic region-aware optimization, which utilizes regional-based semantic information to supervise the training, particularly promoting the reconstruction quality and semantic smoothness. We conduct comprehensive experiments on two real-world surgical datasets to demonstrate the superiority of SurgTPGS over state-of-the-art methods, highlighting its potential to revolutionize surgical practices. SurgTPGS paves the way for developing next-generation intelligent surgical systems by enhancing surgical precision and safety. Our code is available at: https://github.com/lastbasket/SurgTPGS.