标题： AraSTEM：评估大语言模型在STEM学科知识方面的原生阿拉伯语多项选择题基准 显示英文标题

标题： AraSTEM: A Native Arabic Multiple Choice Question Benchmark for Evaluating LLMs Knowledge In STEM Subjects

摘要： 大型语言模型（LLMs）表现出显著的能力，不仅在生成类似人类的文本方面，而且在获取知识方面也是如此。 这突显了需要超越典型的自然语言处理下游基准，并评估LLMs的各个方面，包括知识和推理。 已经开发了许多基准来评估LLMs的知识，但它们主要集中在英语语言上。 鉴于许多LLMs是多语言的，仅依赖于英语知识的评估是不够的。 为了解决这个问题，我们引入了AraSTEM，这是一个新的阿拉伯语多项选择题数据集，旨在评估LLMs在STEM学科中的知识。 该数据集涵盖了不同层次的各种主题，要求模型展示对科学阿拉伯语的深入理解，以实现高准确性。 我们的研究结果表明，各种大小的公开可用模型在这个数据集上遇到困难，并强调了需要更多本地化语言模型的必要性。 该数据集可在Hugging Face上免费访问。 显示英文摘要

摘要： Large Language Models (LLMs) have shown remarkable capabilities, not only in generating human-like text, but also in acquiring knowledge. This highlights the need to go beyond the typical Natural Language Processing downstream benchmarks and asses the various aspects of LLMs including knowledge and reasoning. Numerous benchmarks have been developed to evaluate LLMs knowledge, but they predominantly focus on the English language. Given that many LLMs are multilingual, relying solely on benchmarking English knowledge is insufficient. To address this issue, we introduce AraSTEM, a new Arabic multiple-choice question dataset aimed at evaluating LLMs knowledge in STEM subjects. The dataset spans a range of topics at different levels which requires models to demonstrate a deep understanding of scientific Arabic in order to achieve high accuracy. Our findings show that publicly available models of varying sizes struggle with this dataset, and underscores the need for more localized language models. The dataset is freely accessible on Hugging Face.