标题： 从课程到技能：评估大型语言模型在课程分析中的表现 显示英文标题

标题： From Course to Skill: Evaluating LLM Performance in Curricular Analytics

摘要： 课程分析（CA）——对课程数据进行系统性分析以指导项目和课程改进——成为一种越来越有价值的工具，帮助机构使学术课程与不断变化的社会和经济需求保持一致。 大型语言模型（LLMs）在处理大规模非结构化课程数据方面很有前景，但LLMs执行课程分析任务的可靠性仍不确定。 本文中，我们基于LLMs或传统NLP方法，系统地评估了四种文本对齐策略用于技能提取，这是课程分析中的核心任务。 使用400份不同类型课程文档的分层样本和人机协作评估框架，我们发现检索增强生成（RAG）是在所有类型的课程文档中表现最佳的策略，而零样本提示在大多数情况下表现不如传统的NLP方法。 我们的研究结果突显了LLMs在分析简短抽象课程文档方面的潜力，但也揭示了其性能可能因模型选择和提示策略的不同而显著变化。 这强调了在大规模部署LLMs策略之前仔细评估其性能的重要性。 显示英文摘要

摘要： Curricular analytics (CA) -- systematic analysis of curricula data to inform program and course refinement -- becomes an increasingly valuable tool to help institutions align academic offerings with evolving societal and economic demands. Large language models (LLMs) are promising for handling large-scale, unstructured curriculum data, but it remains uncertain how reliably LLMs can perform CA tasks. In this paper, we systematically evaluate four text alignment strategies based on LLMs or traditional NLP methods for skill extraction, a core task in CA. Using a stratified sample of 400 curriculum documents of different types and a human-LLM collaborative evaluation framework, we find that retrieval-augmented generation (RAG) is the top-performing strategy across all types of curriculum documents, while zero-shot prompting performs worse than traditional NLP methods in most cases. Our findings highlight the promise of LLMs in analyzing brief and abstract curriculum documents, but also reveal that their performance can vary significantly depending on model selection and prompting strategies. This underscores the importance of carefully evaluating the performance of LLM-based strategies before large-scale deployment.