标题： 在开放回答评估中使用大型语言模型生成和蒸馏来增强人工标注的训练数据 显示英文标题

标题： Augmenting Human-Annotated Training Data with Large Language Model Generation and Distillation in Open-Response Assessment

摘要： 大型语言模型（LLMs）如GPT-4o可以帮助以低成本和规模自动化文本分类任务。 然而，人们对LLM输出的有效性和可靠性存在重大担忧。 相比之下，人工编码通常更可靠，但大规模获取成本较高。 在本研究中，我们提出了一种混合解决方案，以利用两者的优势。 我们将人工编码的数据和合成的LLM生成数据结合起来，微调一个经典的机器学习分类器，并将两者提炼到一个较小的BERT模型中。 我们在一个人工编码的测试集上评估我们的方法，作为衡量LLM输出质量的有效性指标。 在三个实验中，我们系统地改变LLM生成样本的数量、多样性和一致性，这些由LLM调优的最佳实践所指导。 我们的研究结果表明，用合成样本扩充数据集可以提高分类器的性能，最佳结果在80%合成数据与20%人工编码数据的比例下实现。 较低的温度设置0.3，对应于LLM生成中的较少变异性，产生了更稳定的改进，但也限制了模型从增强样本中学习的能力。 相反，较高的温度设置（0.7及以上）引入了性能估计中的更大变异性，并且有时导致性能下降。 因此， LLM可能会生成更一致的输出，使分类器过拟合到早期阶段，或者生成更多样化的输出，这可能会因为与预测任务无关的信息而损害模型性能。 过滤掉不一致的合成样本并没有提高性能。 我们得出结论，将人工数据和LLM生成的数据整合起来以改进评估中的文本分类模型，提供了一个可扩展的解决方案，该方案利用了人工编码的准确性和LLM输出的多样性。 显示英文摘要

摘要： Large Language Models (LLMs) like GPT-4o can help automate text classification tasks at low cost and scale. However, there are major concerns about the validity and reliability of LLM outputs. By contrast, human coding is generally more reliable but expensive to procure at scale. In this study, we propose a hybrid solution to leverage the strengths of both. We combine human-coded data and synthetic LLM-produced data to fine-tune a classical machine learning classifier, distilling both into a smaller BERT model. We evaluate our method on a human-coded test set as a validity measure for LLM output quality. In three experiments, we systematically vary LLM-generated samples' size, variety, and consistency, informed by best practices in LLM tuning. Our findings indicate that augmenting datasets with synthetic samples improves classifier performance, with optimal results achieved at an 80% synthetic to 20% human-coded data ratio. Lower temperature settings of 0.3, corresponding to less variability in LLM generations, produced more stable improvements but also limited model learning from augmented samples. In contrast, higher temperature settings (0.7 and above) introduced greater variability in performance estimates and, at times, lower performance. Hence, LLMs may produce more uniform output that classifiers overfit to earlier or produce more diverse output that runs the risk of deteriorating model performance through information irrelevant to the prediction task. Filtering out inconsistent synthetic samples did not enhance performance. We conclude that integrating human and LLM-generated data to improve text classification models in assessment offers a scalable solution that leverages both the accuracy of human coding and the variety of LLM outputs.