标题： ShapeLLM-Omni：一种用于3D生成和理解的原生多模态大型语言模型 显示英文标题

标题： ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understanding

摘要： 近期，ChatGPT-4o强大的文本到图像能力引发了对原生多模态大型语言模型的日益关注。然而，其多模态能力仍局限于图像和文本。除了图像之外，理解和生成3D内容的能力同样至关重要。为填补这一空白，我们提出了ShapeLLM-Omni——一种能够理解并生成任意顺序3D资产和文本的原生3D大型语言模型。 首先，我们训练了一种3D向量量化变分自编码器（VQVAE），它将3D物体映射到离散潜在空间，以实现高效且准确的形状表示与重建。 在此基础上，我们创新性地构建了一个名为3D-Alpaca的大规模连续训练数据集，涵盖了生成、理解与编辑，从而为未来的科研和训练提供了丰富的资源。 最后，在3D-Alpaca数据集上对Qwen-2.5-vl-7B-Instruct模型进行了基于指令的训练。 我们的工作为扩展具备基本3D能力的多模态模型提供了一种有效的尝试，这有助于未来3D原生人工智能的研究。 项目页面：https://github.com/JAMESYJL/ShapeLLM-Omni 显示英文摘要

摘要： Recently, the powerful text-to-image capabilities of ChatGPT-4o have led to growing appreciation for native multimodal large language models. However, its multimodal capabilities remain confined to images and text. Yet beyond images, the ability to understand and generate 3D content is equally crucial. To address this gap, we propose ShapeLLM-Omni-a native 3D large language model capable of understanding and generating 3D assets and text in any sequence. First, we train a 3D vector-quantized variational autoencoder (VQVAE), which maps 3D objects into a discrete latent space to achieve efficient and accurate shape representation and reconstruction. Building upon the 3D-aware discrete tokens, we innovatively construct a large-scale continuous training dataset named 3D-Alpaca, encompassing generation, comprehension, and editing, thus providing rich resources for future research and training. Finally, by performing instruction-based training of the Qwen-2.5-vl-7B-Instruct model on the 3D-Alpaca dataset. Our work provides an effective attempt at extending multimodal models with basic 3D capabilities, which contributes to future research in 3D-native AI. Project page: https://github.com/JAMESYJL/ShapeLLM-Omni