标题： 从课程到技能：LLM在课程分析中的性能评估 显示英文标题

标题： From Course to Skill: Evaluating LLM Performance in Curricular Analytics

摘要： 课程分析（CA）——系统分析课程数据以指导项目和课程的改进——成为一种越来越有价值的工具，帮助机构使学术课程与不断变化的社会和经济需求保持一致。大型语言模型（LLMs）在处理大规模、非结构化的课程数据方面很有前景，但LLMs执行课程分析任务的可靠性仍然不确定。在本文中，我们系统地评估了四种基于LLMs或传统NLP方法的文本对齐策略，用于技能提取，这是课程分析的核心任务。使用由不同类型课程文档组成的分层样本和人机协作评估框架，我们发现检索增强生成（RAG）是在所有类型的课程文档中表现最好的策略，而零样本提示在大多数情况下比传统NLP方法表现更差。我们的研究结果突显了LLMs在分析简短且抽象的课程文档方面的潜力，但也揭示了它们的表现可能因模型选择和提示策略的不同而显著不同。这强调了在大规模部署前仔细评估基于LLMs的策略性能的重要性。 显示英文摘要

摘要： Curricular analytics (CA) -- systematic analysis of curricula data to inform program and course refinement -- becomes an increasingly valuable tool to help institutions align academic offerings with evolving societal and economic demands. Large language models (LLMs) are promising for handling large-scale, unstructured curriculum data, but it remains uncertain how reliably LLMs can perform CA tasks. In this paper, we systematically evaluate four text alignment strategies based on LLMs or traditional NLP methods for skill extraction, a core task in CA. Using a stratified sample of 400 curriculum documents of different types and a human-LLM collaborative evaluation framework, we find that retrieval-augmented generation (RAG) to be the top-performing strategy across all types of curriculum documents, while zero-shot prompting performs worse than traditional NLP methods in most cases. Our findings highlight the promise of LLMs in analyzing brief and abstract curriculum documents, but also reveal that their performance can vary significantly depending on model selection and prompting strategies. This underscores the importance of carefully evaluating the performance of LLM-based strategies before large-scale deployment.