Title: Benchmarking Large Language Models on Homework Assessment in Circuit Analysis Show Chinese title

Title: 在电路分析作业评估中大型语言模型的基准测试

Abstract: Large language models (LLMs) have the potential to revolutionize various fields, including code development, robotics, finance, and education, due to their extensive prior knowledge and rapid advancements. This paper investigates how LLMs can be leveraged in engineering education. Specifically, we benchmark the capabilities of different LLMs, including GPT-3.5 Turbo, GPT-4o, and Llama 3 70B, in assessing homework for an undergraduate-level circuit analysis course. We have developed a novel dataset consisting of official reference solutions and real student solutions to problems from various topics in circuit analysis. To overcome the limitations of image recognition in current state-of-the-art LLMs, the solutions in the dataset are converted to LaTeX format. Using this dataset, a prompt template is designed to test five metrics of student solutions: completeness, method, final answer, arithmetic error, and units. The results show that GPT-4o and Llama 3 70B perform significantly better than GPT-3.5 Turbo across all five metrics, with GPT-4o and Llama 3 70B each having distinct advantages in different evaluation aspects. Additionally, we present insights into the limitations of current LLMs in several aspects of circuit analysis. Given the paramount importance of ensuring reliability in LLM-generated homework assessment to avoid misleading students, our results establish benchmarks and offer valuable insights for the development of a reliable, personalized tutor for circuit analysis -- a focus of our future work. Furthermore, the proposed evaluation methods can be generalized to a broader range of courses for engineering education in the future. Show Chinese abstract

Abstract: 大型语言模型（LLMs）由于其广泛的知识和快速进步，有可能彻底改变包括代码开发、机器人技术、金融和教育在内的各个领域。 本文研究了如何在工程教育中利用LLMs。 具体来说，我们评估了GPT-3.5 Turbo、GPT-4o和Llama 3 70B等不同LLMs在评估本科电路分析课程作业中的能力。 我们开发了一个新的数据集，其中包括电路分析各主题的官方参考答案和真实学生答案。 为克服当前最先进的LLMs中图像识别的局限性，数据集中的解决方案被转换为LaTeX格式。 使用这个数据集，设计了一个提示模板来测试学生解答的五个指标：完整性、方法、最终答案、算术错误和单位。 结果显示，GPT-4o和Llama 3 70B在所有五个指标上都显著优于GPT-3.5 Turbo，在不同的评估方面各有优势。 此外，我们还展示了当前LLMs在电路分析几个方面的局限性。 鉴于确保LLM生成的作业评估可靠性以避免误导学生的极端重要性，我们的结果为开发可靠、个性化的电路分析导师建立了基准并提供了有价值的见解——这是我们未来工作的重点。 此外，所提出的评估方法可以推广到工程教育的更广泛的课程中。