标题： 基于条件共形真实性的检索增强生成响应质量评估 显示英文标题

标题： Response Quality Assessment for Retrieval-Augmented Generation via Conditional Conformal Factuality

摘要： 现有关于检索增强生成（RAG）的研究主要集中在提高整体问答准确性，常常忽视生成响应中子主张的质量。 最近试图提高RAG可信度的方法，例如通过自动评估指标，缺乏概率保证或需要真实答案。 为了解决这些限制，我们提出了Conformal-RAG，这是一个受最近在大型语言模型（LLMs）上应用的符合预测（CP）启发的新框架。 Conformal-RAG利用CP和RAG机制中的内部信息，对响应质量提供统计保证。 它确保跨多个子领域的组条件覆盖，而无需手动标记符合集，使其适用于复杂的RAG应用。 与现有的RAG自动评估方法相比，Conformal-RAG对精炼子主张的质量提供了统计保证，确保响应可靠性而无需真实答案。 此外，我们的实验表明，通过利用RAG系统的信息，Conformal-RAG相比直接将CP应用于LLMs，保留了多达60%的高质量子主张，同时保持相同的可靠性保证。 显示英文摘要

摘要： Existing research on Retrieval-Augmented Generation (RAG) primarily focuses on improving overall question-answering accuracy, often overlooking the quality of sub-claims within generated responses. Recent methods that attempt to improve RAG trustworthiness, such as through auto-evaluation metrics, lack probabilistic guarantees or require ground truth answers. To address these limitations, we propose Conformal-RAG, a novel framework inspired by recent applications of conformal prediction (CP) on large language models (LLMs). Conformal-RAG leverages CP and internal information from the RAG mechanism to offer statistical guarantees on response quality. It ensures group-conditional coverage spanning multiple sub-domains without requiring manual labelling of conformal sets, making it suitable for complex RAG applications. Compared to existing RAG auto-evaluation methods, Conformal-RAG offers statistical guarantees on the quality of refined sub-claims, ensuring response reliability without the need for ground truth answers. Additionally, our experiments demonstrate that by leveraging information from the RAG system, Conformal-RAG retains up to 60\% more high-quality sub-claims from the response compared to direct applications of CP to LLMs, while maintaining the same reliability guarantee.