标题： 构造响应评分的有效性论证使用生成式人工智能应用 显示英文标题

标题： Validity Arguments For Constructed Response Scoring Using Generative Artificial Intelligence Applications

摘要： 大型语言模型和生成式人工智能（AI）能力的迅速发展，使得它们在高风险测试环境中的广泛应用变得更加可能。 在构造性回答评分中使用生成式AI特别有吸引力，因为它减少了传统AI评分中手工设计特征所需的工作量，甚至可能超越这些方法。 本文的目的是突出基于特征的AI和生成式AI在构造性回答评分系统中的应用差异，并提出一套最佳实践，以收集有效性证据，支持使用和解释基于生成式AI评分系统的构造性回答分数。 我们比较了使用人工评分、基于特征的自然语言处理AI评分引擎和生成式AI的评分系统所需的有效性证据。 由于生成式AI缺乏透明度以及其他独特的担忧，如一致性问题，生成式AI情境下所需的有效性证据比基于特征的NLP评分情境下更为广泛。 标准化测试中的构造性回答分数数据展示了不同类型评分系统有效性证据的收集情况，并突出了在为这些分数构建有效性论点时的诸多复杂性和考虑因素。 此外，我们讨论了AI分数的评估可能需要考虑一种结合多个AI分数（来自不同来源）的贡献性评分方法，在没有人工评分的情况下，这种方法将更全面地覆盖构念。 显示英文摘要

摘要： The rapid advancements in large language models and generative artificial intelligence (AI) capabilities are making their broad application in the high-stakes testing context more likely. Use of generative AI in the scoring of constructed responses is particularly appealing because it reduces the effort required for handcrafting features in traditional AI scoring and might even outperform those methods. The purpose of this paper is to highlight the differences in the feature-based and generative AI applications in constructed response scoring systems and propose a set of best practices for the collection of validity evidence to support the use and interpretation of constructed response scores from scoring systems using generative AI. We compare the validity evidence needed in scoring systems using human ratings, feature-based natural language processing AI scoring engines, and generative AI. The evidence needed in the generative AI context is more extensive than in the feature-based NLP scoring context because of the lack of transparency and other concerns unique to generative AI such as consistency. Constructed response score data from standardized tests demonstrate the collection of validity evidence for different types of scoring systems and highlights the numerous complexities and considerations when making a validity argument for these scores. In addition, we discuss how the evaluation of AI scores might include a consideration of how a contributory scoring approach combining multiple AI scores (from different sources) will cover more of the construct in the absence of human ratings.