标题： 评估视觉语言动作启用机器人的不确定性与质量 显示英文标题

标题： Evaluating Uncertainty and Quality of Visual Language Action-enabled Robots

摘要： 视觉语言动作（VLA）模型是人工智能（AI）系统的一个多模态类别，它结合了视觉感知、自然语言理解和动作规划，使代理能够解释其环境、理解指令并自主执行具身任务。最近，该领域取得了显著进展。这些模型通常通过任务成功率进行评估，这无法捕捉任务执行的质量以及模型对其决策的置信度。在本文中，我们提出了八个针对机器人操作任务的VLA模型专门设计的不确定性指标和五个质量指标。我们通过一项涉及三个最先进的VLA模型在四个代表性机器人操作任务中成功执行的908次任务的大规模实证研究来评估它们的有效性。人类领域专家手动标记了任务质量，使我们能够分析所提出的指标与专家判断之间的相关性。结果表明，一些指标与人类评估显示出中等至强相关性，突显了它们在评估任务质量和模型置信度方面的实用性。此外，我们发现某些指标可以区分失败任务中的高质量、中质量和低质量执行，这在测试预言不存在时可能很有意思。我们的研究结果挑战了目前仅依赖二进制成功比率的评估实践，并为改进VLA支持的机器人系统的实时监控和自适应增强铺平了道路。 显示英文摘要

摘要： Visual Language Action (VLA) models are a multi-modal class of Artificial Intelligence (AI) systems that integrate visual perception, natural language understanding, and action planning to enable agents to interpret their environment, comprehend instructions, and perform embodied tasks autonomously. Recently, significant progress has been made to advance this field. These kinds of models are typically evaluated through task success rates, which fail to capture the quality of task execution and the mode's confidence in its decisions. In this paper, we propose eight uncertainty metrics and five quality metrics specifically designed for VLA models for robotic manipulation tasks. We assess their effectiveness through a large-scale empirical study involving 908 successful task executions from three state-of-the-art VLA models across four representative robotic manipulation tasks. Human domain experts manually labeled task quality, allowing us to analyze the correlation between our proposed metrics and expert judgments. The results reveal that several metrics show moderate to strong correlation with human assessments, highlighting their utility for evaluating task quality and model confidence. Furthermore, we found that some of the metrics can discriminate between high-, medium-, and low-quality executions from unsuccessful tasks, which can be interesting when test oracles are not available. Our findings challenge the adequacy of current evaluation practices that rely solely on binary success rates and pave the way for improved real-time monitoring and adaptive enhancement of VLA-enabled robotic systems.