Title: Can LLMs Identify Gaps and Misconceptions in Students' Code Explanations? Show Chinese title

Title: LLMs能否识别学生代码解释中的漏洞和误解？

Abstract: This paper investigates various approaches using Large Language Models (LLMs) to identify gaps and misconceptions in students' self-explanations of specific instructional material, in our case explanations of code examples. This research is a part of our larger effort to automate the assessment of students' freely generated responses, focusing specifically on their self-explanations of code examples during activities related to code comprehension. In this work, we experiment with zero-shot prompting, Supervised Fine-Tuning (SFT), and preference alignment of LLMs to identify gaps in students' self-explanation. With simple prompting, GPT-4 consistently outperformed LLaMA3 and Mistral in identifying gaps and misconceptions, as confirmed by human evaluations. Additionally, our results suggest that fine-tuned large language models are more effective at identifying gaps in students' explanations compared to zero-shot and few-shot prompting techniques. Furthermore, our findings show that the preference optimization approach using Odds Ratio Preference Optimization (ORPO) outperforms SFT in identifying gaps and misconceptions in students' code explanations. Show Chinese abstract

Abstract: 本文研究了使用大型语言模型（LLMs）的各种方法，以识别学生在特定教学材料自我解释中的差距和误解，我们的情况是代码示例的解释。 这项研究是我们更大努力的一部分，旨在自动化评估学生自由生成的回答，特别关注他们在与代码理解相关的活动中的自我解释。 在这项工作中，我们尝试了零样本提示、监督微调（SFT）和大型语言模型的偏好对齐，以识别学生自我解释中的差距。 通过简单的提示，GPT-4在识别差距和误解方面始终优于LLaMA3和Mistral，这经过人工评估得到了确认。 此外，我们的结果表明，微调后的大型语言模型在识别学生解释中的差距方面比零样本和少样本提示技术更有效。 此外，我们的研究结果表明，使用几率比偏好优化（ORPO）的偏好优化方法在识别学生代码解释中的差距和误解方面优于SFT。