标题： 现实世界摘要：当评估达到其极限时 显示英文标题

标题： Real-World Summarization: When Evaluation Reaches Its Limits

摘要： 我们考察了在酒店亮点上下文中对输入数据的忠实性评估：简短的LLM生成摘要，这些摘要捕捉了住宿的独特特征。 通过涉及分类错误评估和跨度级别注释的人类评估活动，我们比较了传统指标、可训练方法和LLM作为裁判的方法。 我们的研究结果表明，简单的指标如词重叠与人类判断相关性出人意料地好（斯皮尔曼等级相关系数为0.63），在应用于领域外数据时，通常优于更复杂的方法。 我们进一步证明，虽然LLM可以生成高质量的亮点，但它们在评估方面不可靠，因为它们往往严重低估或高估注释。 我们对现实世界业务影响的分析显示，错误且无法验证的信息构成了最大的风险。 我们还强调了众包评估中的挑战。 显示英文摘要

摘要： We examine evaluation of faithfulness to input data in the context of hotel highlights: brief LLM-generated summaries that capture unique features of accommodations. Through human evaluation campaigns involving categorical error assessment and span-level annotation, we compare traditional metrics, trainable methods, and LLM-as-a-judge approaches. Our findings reveal that simpler metrics like word overlap correlate surprisingly well with human judgments (Spearman correlation rank of 0.63), often outperforming more complex methods when applied to out-of-domain data. We further demonstrate that while LLMs can generate high-quality highlights, they prove unreliable for evaluation as they tend to severely under- or over-annotate. Our analysis of real-world business impacts shows incorrect and non-checkable information pose the greatest risks. We also highlight challenges in crowdsourced evaluations.