标题： MapEval：基于地图的大型基础模型地理空间推理评估 显示英文标题

标题： MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models

摘要： 基础模型领域的最新进展提升了自主工具使用和推理能力，但它们在基于地图的推理方面的潜力仍未被充分探索。 为了解决这一问题，我们引入了 MapEval，这是一个旨在通过涵盖180个城市和54个国家的700道多项选择题来评估基础模型在三种不同任务——文本推理、基于API的推理和视觉推理——上的表现的基准测试框架，涉及空间关系、导航、旅行规划以及与现实世界地图的交互。 与专注于简单位置查询的先前基准测试不同，MapEval 要求模型处理长上下文推理、API交互和视觉地图分析，使其成为地理空间人工智能最全面的评估框架。 在对包括Claude-3.5-Sonnet、GPT-4o和Gemini-1.5-Pro在内的30个基础模型进行评估时，没有一个模型的准确率超过67%，开源模型的表现明显更差，所有模型的表现均落后于人类水平20%以上。 这些结果揭示了空间推理中的关键差距，因为模型在距离、方向、路线规划和特定地点推理方面存在困难，突显了开发更好的地理空间人工智能以弥合基础模型与现实世界导航之间差距的需求。 所有资源均可在以下网址获取：https://mapeval.github.io/. 显示英文摘要

摘要： Recent advancements in foundation models have improved autonomous tool usage and reasoning, but their capabilities in map-based reasoning remain underexplored. To address this, we introduce MapEval, a benchmark designed to assess foundation models across three distinct tasks - textual, API-based, and visual reasoning - through 700 multiple-choice questions spanning 180 cities and 54 countries, covering spatial relationships, navigation, travel planning, and real-world map interactions. Unlike prior benchmarks that focus on simple location queries, MapEval requires models to handle long-context reasoning, API interactions, and visual map analysis, making it the most comprehensive evaluation framework for geospatial AI. On evaluation of 30 foundation models, including Claude-3.5-Sonnet, GPT-4o, and Gemini-1.5-Pro, none surpass 67% accuracy, with open-source models performing significantly worse and all models lagging over 20% behind human performance. These results expose critical gaps in spatial inference, as models struggle with distances, directions, route planning, and place-specific reasoning, highlighting the need for better geospatial AI to bridge the gap between foundation models and real-world navigation. All the resources are available at: https://mapeval.github.io/.