标题： 理解空间是火箭科学——只有顶级推理模型才能解决空间理解任务 显示英文标题

标题： Understanding Space Is Rocket Science -- Only Top Reasoning Models Can Solve Spatial Understanding Tasks

摘要： 我们提出RocketScience，一个开源的对比视觉语言模型基准，用于测试空间关系理解能力。 它由完全新的真实世界图像-文本对组成，主要涵盖相对空间理解和物体顺序。 该基准设计为对人类非常容易，但对当前一代的视觉语言模型来说很难，并且通过实证得到了验证。 我们的结果表明，开源和前沿商业视觉语言模型在空间关系理解方面存在显著不足，而推理模型的表现却出人意料地高。 此外，我们进行了解耦分析，以分离基于思维链的模型中物体定位和空间推理的贡献，并发现该基准上的性能受限于空间推理，而非物体定位能力。 我们以CC-BY-4.0许可证发布数据集，并在以下链接提供评估代码：https://github.com/nilshoehing/rocketscience 显示英文摘要

摘要： We propose RocketScience, an open-source contrastive VLM benchmark that tests for spatial relation understanding. It is comprised of entirely new real-world image-text pairs covering mostly relative spatial understanding and the order of objects. The benchmark is designed to be very easy for humans and hard for the current generation of VLMs, and this is empirically verified. Our results show a striking lack of spatial relation understanding in open source and frontier commercial VLMs and a surprisingly high performance of reasoning models. Additionally, we perform a disentanglement analysis to separate the contributions of object localization and spatial reasoning in chain-of-thought-based models and find that the performance on the benchmark is bottlenecked by spatial reasoning and not object localization capabilities. We release the dataset with a CC-BY-4.0 license and make the evaluation code available at: https://github.com/nilshoehing/rocketscience