标题： 用于增强现实中的多模态训练助手的细粒度视觉-语言建模 显示英文标题

标题： Fine-Grained Vision-Language Modeling for Multimodal Training Assistants in Augmented Reality

摘要： 视觉-语言模型（VLMs）对于使人工智能驱动的智能助手在多模态环境中进行解释和推理至关重要。 然而，它们在增强现实（AR）培训中的应用仍基本未被探索。 在本工作中，我们引入了一个针对AR培训的全面数据集，包含系统化的视觉-语言任务，并在该数据集上评估了九种最先进的VLMs。 我们的结果表明，即使是先进的模型，包括GPT-4o，在细粒度组装任务中也表现出困难，在状态检测上的最大F1分数仅为40.54%。 这些发现突显了对增强的数据集、基准测试和进一步研究的需求，以提高细粒度的视觉-语言对齐能力。 除了技术贡献外，我们的工作还具有更广泛的社会影响，特别是在为盲人和视力受损用户提供平等的人工智能驱动学习机会方面。 我们提供了所有相关资源，包括数据集、源代码和评估结果，以支持研究社区。 显示英文摘要

摘要： Vision-language models (VLMs) are essential for enabling AI-powered smart assistants to interpret and reason in multimodal environments. However, their application in augmented reality (AR) training remains largely unexplored. In this work, we introduce a comprehensive dataset tailored for AR training, featuring systematized vision-language tasks, and evaluate nine state-of-the-art VLMs on it. Our results reveal that even advanced models, including GPT-4o, struggle with fine-grained assembly tasks, achieving a maximum F1 score of just 40.54% on state detection. These findings highlight the demand for enhanced datasets, benchmarks, and further research to improve fine-grained vision-language alignment. Beyond technical contributions, our work has broader social implications, particularly in empowering blind and visually impaired users with equitable access to AI-driven learning opportunities. We provide all related resources, including the dataset, source code, and evaluation results, to support the research community.