Title: Evaluating the Symbol Binding Ability of Large Language Models for Multiple-Choice Questions in Vietnamese General Education Show Chinese title

Title: 评估大型语言模型在越南通用教育多项选择题中的符号绑定能力

Abstract: In this paper, we evaluate the ability of large language models (LLMs) to perform multiple choice symbol binding (MCSB) for multiple choice question answering (MCQA) tasks in zero-shot, one-shot, and few-shot settings. We focus on Vietnamese, with fewer challenging MCQA datasets than in English. The two existing datasets, ViMMRC 1.0 and ViMMRC 2.0, focus on literature. Recent research in Vietnamese natural language processing (NLP) has focused on the Vietnamese National High School Graduation Examination (VNHSGE) from 2019 to 2023 to evaluate ChatGPT. However, these studies have mainly focused on how ChatGPT solves the VNHSGE step by step. We aim to create a novel and high-quality dataset by providing structured guidelines for typing LaTeX formulas for mathematics, physics, chemistry, and biology. This dataset can be used to evaluate the MCSB ability of LLMs and smaller language models (LMs) because it is typed in a strict LaTeX style. We focus on predicting the character (A, B, C, or D) that is the most likely answer to a question, given the context of the question. Our evaluation of six well-known LLMs, namely BLOOMZ-7.1B-MT, LLaMA-2-7B, LLaMA-2-70B, GPT-3, GPT-3.5, and GPT-4.0, on the ViMMRC 1.0 and ViMMRC 2.0 benchmarks and our proposed dataset shows promising results on the MCSB ability of LLMs for Vietnamese. The dataset is available for research purposes only. Show Chinese abstract

Abstract: 在本文中，我们评估了大型语言模型（LLMs）在零样本、单样本和少量样本设置下执行多项选择符号绑定（MCSB）以进行多项选择问题回答（MCQA）任务的能力。 我们专注于越南语，其具有挑战性的MCQA数据集比英语少。 现有的两个数据集，ViMMRC 1.0和ViMMRC 2.0，侧重于文学。 最近的越南语自然语言处理（NLP）研究集中在2019年至2023年的越南国家高中毕业考试（VNHSGE）上，以评估ChatGPT。 然而，这些研究主要关注ChatGPT如何逐步解决VNHSGE。 我们旨在通过为数学、物理、化学和生物的LaTeX公式输入提供结构化指南来创建一个新颖且高质量的数据集。 由于该数据集采用严格的LaTeX格式输入，因此可以用于评估LLMs和较小的语言模型（LMs）的MCSB能力。 我们专注于在给定问题上下文的情况下预测最可能是问题答案的字符（A、B、C或D）。 我们在ViMMRC 1.0和ViMMRC 2.0基准以及我们提出的数据集上对六种著名的LLMs，即BLOOMZ-7.1B-MT、LLaMA-2-7B、LLaMA-2-70B、GPT-3、GPT-3.5和GPT-4.0进行了评估，结果显示LLMs在越南语的MCSB能力方面具有良好的前景。 该数据集仅限于研究目的。