标题： CaseSumm：来自美国最高法院意见的长上下文摘要的大规模数据集 显示英文标题

标题： CaseSumm: A Large-Scale Dataset for Long-Context Summarization from U.S. Supreme Court Opinions

摘要： 本文介绍了 CaseSumm，这是一个针对法律领域长上下文摘要的新数据集，旨在满足对更长、更复杂数据集用于摘要评估的需求。 我们收集了 25,600 篇美国最高法院 (SCOTUS) 意见及其官方摘要，这些摘要被称为“概要”。 我们的数据集是最大的公开可用的法律案件摘要数据集，并且是首个包括追溯到 1815 年的 SCOTUS 决策摘要的数据集。 我们还通过自动指标和专家人工评估全面评估了大型语言模型生成的摘要，揭示了这些评估方法之间的差异。 我们的评估显示，Mistral 7b，一个较小的开源模型，在大多数自动指标上优于更大的模型，并成功生成了类似概要的摘要。 相比之下，人类专家标注员指出 Mistral 摘要包含幻觉。 标注员一致认为 GPT-4 摘要更清晰，并表现出更高的敏感性和特异性。 此外，我们发现基于大型语言模型的评估与传统自动指标相比，并未显示出更高的与人类评估的相关性。 此外，我们的分析确定了生成摘要中的特定幻觉，包括先例引用错误和案件事实的误传。 这些发现展示了当前自动评估方法在法律摘要方面的局限性，并强调了人工评估在评估摘要质量方面的重要作用，特别是在复杂且高风险的领域。 CaseSumm 可在 https://huggingface.co/datasets/ChicagoHAI/CaseSumm 获取。 显示英文摘要

摘要： This paper introduces CaseSumm, a novel dataset for long-context summarization in the legal domain that addresses the need for longer and more complex datasets for summarization evaluation. We collect 25.6K U.S. Supreme Court (SCOTUS) opinions and their official summaries, known as "syllabuses." Our dataset is the largest open legal case summarization dataset, and is the first to include summaries of SCOTUS decisions dating back to 1815. We also present a comprehensive evaluation of LLM-generated summaries using both automatic metrics and expert human evaluation, revealing discrepancies between these assessment methods. Our evaluation shows Mistral 7b, a smaller open-source model, outperforms larger models on most automatic metrics and successfully generates syllabus-like summaries. In contrast, human expert annotators indicate that Mistral summaries contain hallucinations. The annotators consistently rank GPT-4 summaries as clearer and exhibiting greater sensitivity and specificity. Further, we find that LLM-based evaluations are not more correlated with human evaluations than traditional automatic metrics. Furthermore, our analysis identifies specific hallucinations in generated summaries, including precedent citation errors and misrepresentations of case facts. These findings demonstrate the limitations of current automatic evaluation methods for legal summarization and highlight the critical role of human evaluation in assessing summary quality, particularly in complex, high-stakes domains. CaseSumm is available at https://huggingface.co/datasets/ChicagoHAI/CaseSumm