标题： 衡量大型语言模型标注新闻来源的能力 显示英文标题

标题： Measuring Large Language Models Capacity to Annotate Journalistic Sourcing

摘要： 自2022年底ChatGPT发布以来，大型语言模型及其评估在学术研究和工业界一直受到持续的讨论和评估。在法律、医学和数学等多个领域已经开发了相应的场景和基准（Bommasani等，2023），并且对模型变体进行持续评估。一个尚未得到足够场景开发关注的领域是新闻业，特别是新闻来源和伦理问题。新闻业是民主中至关重要的真相判定功能（Vincent，2023），而新闻来源是所有原创新闻产出的关键支柱。评估大型语言模型在不同新闻故事中识别和标注新闻来源信号以及记者如何为其提供依据的能力是一个关键场景，值得采用基准方法。这为构建自动化系统以对比更加透明和伦理严谨的新闻形式与日常新闻提供了潜力。在本文中，我们提出一个场景，用于评估大型语言模型在五类新闻来源标识和标注方面的表现，该分类体系受到新闻研究的启发（Gans，2004）。我们提供了使用案例、我们的数据集和指标，作为系统基准测试的第一步。我们的准确度发现表明，基于大型语言模型的方法在识别故事中的所有有据声明以及匹配来源类型方面还有更多需要改进的地方。更困难的任务是识别来源依据。 显示英文摘要

摘要： Since the launch of ChatGPT in late 2022, the capacities of Large Language Models and their evaluation have been in constant discussion and evaluation both in academic research and in the industry. Scenarios and benchmarks have been developed in several areas such as law, medicine and math (Bommasani et al., 2023) and there is continuous evaluation of model variants. One area that has not received sufficient scenario development attention is journalism, and in particular journalistic sourcing and ethics. Journalism is a crucial truth-determination function in democracy (Vincent, 2023), and sourcing is a crucial pillar to all original journalistic output. Evaluating the capacities of LLMs to annotate stories for the different signals of sourcing and how reporters justify them is a crucial scenario that warrants a benchmark approach. It offers potential to build automated systems to contrast more transparent and ethically rigorous forms of journalism with everyday fare. In this paper we lay out a scenario to evaluate LLM performance on identifying and annotating sourcing in news stories on a five-category schema inspired from journalism studies (Gans, 2004). We offer the use case, our dataset and metrics and as the first step towards systematic benchmarking. Our accuracy findings indicate LLM-based approaches have more catching to do in identifying all the sourced statements in a story, and equally, in matching the type of sources. An even harder task is spotting source justifications.