标题： 用于大型班级短答案任务的NLP辅助形成性评估反馈的基础 显示英文标题

标题： Foundations for NLP-assisted formative assessment feedback for short-answer tasks in large-enrollment classes

摘要： 研究建议“写作学习”任务可以提高学习成果，然而，在大班教学中，构造性回答形式的形成性评估方法变得难以管理。 本研究评估了自然语言处理算法以协助这一目标。 1,935名学生完成了六个简答题，这些题目由多位人类评分者根据详细的评分标准和一个算法进行评分。 结果表明，使用二次加权Kappa系数进行评分者对之间的显著一致性（每个QWK > 0.74）以及小组共识（Fleiss Kappa = 0.68）。 此外，还估计了一位评分者在七年前评分的178个回答的内部一致性（QWK = 0.89）。 在评分者之间具有明显一致性的基础上，研究随后试点对回答文本进行聚类分析，以使教师能够通过聚类赋予意义，从而实现可扩展的形成性评估。 显示英文摘要

摘要： Research suggests "write-to-learn" tasks improve learning outcomes, yet constructed-response methods of formative assessment become unwieldy with large class sizes. This study evaluates natural language processing algorithms to assist this aim. Six short-answer tasks completed by 1,935 students were scored by several human raters, using a detailed rubric, and an algorithm. Results indicate substantial inter-rater agreement using quadratic weighted kappa for rater pairs (each QWK > 0.74) and group consensus (Fleiss Kappa = 0.68). Additionally, intra-rater agreement was estimated for one rater who had scored 178 responses seven years prior (QWK = 0.89). With compelling rater agreement, the study then pilots cluster analysis of response text toward enabling instructors to ascribe meaning to clusters as a means for scalable formative assessment.