标题： Absher：评估大型语言模型对沙特方言理解的基准 显示英文标题

标题： Absher: A Benchmark for Evaluating Large Language Models Understanding of Saudi Dialects

摘要： 随着大型语言模型（LLMs）在阿拉伯语自然语言处理应用中变得越来越核心，评估它们对地区方言和文化细微差别的理解至关重要，尤其是在像沙特阿拉伯这样语言多样化的环境中。 本文介绍了\texttt{阿布舍尔}，一个专门设计用于评估LLMs在主要沙特方言中的表现的综合性基准。 \texttt{阿布舍尔}包含超过18,000道选择题，涵盖六个不同的类别：含义、对错、填空、上下文使用、文化解释和地点识别。 这些问题来源于从沙特阿拉伯各地收集的方言词汇、短语和谚语的精选数据集。 我们评估了几种最先进的LLMs，包括多语言和阿拉伯语专用模型。 我们还提供了它们能力与局限性的详细见解。 我们的结果揭示了显著的性能差距，特别是在需要文化推理或上下文理解的任务中。 我们的研究结果突显了迫切需要进行方言感知训练和文化一致的评估方法，以提高LLMs在现实世界阿拉伯语应用中的表现。 显示英文摘要

摘要： As large language models (LLMs) become increasingly central to Arabic NLP applications, evaluating their understanding of regional dialects and cultural nuances is essential, particularly in linguistically diverse settings like Saudi Arabia. This paper introduces \texttt{Absher}, a comprehensive benchmark specifically designed to assess LLMs performance across major Saudi dialects. \texttt{Absher} comprises over 18,000 multiple-choice questions spanning six distinct categories: Meaning, True/False, Fill-in-the-Blank, Contextual Usage, Cultural Interpretation, and Location Recognition. These questions are derived from a curated dataset of dialectal words, phrases, and proverbs sourced from various regions of Saudi Arabia. We evaluate several state-of-the-art LLMs, including multilingual and Arabic-specific models. We also provide detailed insights into their capabilities and limitations. Our results reveal notable performance gaps, particularly in tasks requiring cultural inference or contextual understanding. Our findings highlight the urgent need for dialect-aware training and culturally aligned evaluation methodologies to improve LLMs performance in real-world Arabic applications.