标题： 超越语义熵：利用成对语义相似性提升大语言模型的不确定性量化 显示英文标题

标题： Beyond Semantic Entropy: Boosting LLM Uncertainty Quantification with Pairwise Semantic Similarity

摘要： 大型语言模型（LLMs）中的幻觉可以通过评估模型输出的不确定性来检测，通常使用熵来衡量。 语义熵（SE）通过量化语义聚类级别的不确定性增强了传统熵估计。 然而，随着现代LLMs生成更长的一句话回复，SE的效果减弱了，因为它忽略了两个关键因素：簇内相似性（簇内的扩散）和簇间相似性（簇之间的距离）。 为了解决这些局限性，我们提出了一种受最近邻熵估计启发的简单黑盒不确定性量化方法。 我们的方法还可以通过结合标记概率轻松扩展到白盒设置。 此外，我们提供了理论结果，表明我们的方法概括了语义熵。 广泛的实证结果表明，在两个最新的LLMs（Phi3和Llama3）和三个常见的文本生成任务（问答、文本摘要和机器翻译）上，它比语义熵更有效。 我们的代码可在https://github.com/BigML-CS-UCLA/SNNE获得。 显示英文摘要

摘要： Hallucination in large language models (LLMs) can be detected by assessing the uncertainty of model outputs, typically measured using entropy. Semantic entropy (SE) enhances traditional entropy estimation by quantifying uncertainty at the semantic cluster level. However, as modern LLMs generate longer one-sentence responses, SE becomes less effective because it overlooks two crucial factors: intra-cluster similarity (the spread within a cluster) and inter-cluster similarity (the distance between clusters). To address these limitations, we propose a simple black-box uncertainty quantification method inspired by nearest neighbor estimates of entropy. Our approach can also be easily extended to white-box settings by incorporating token probabilities. Additionally, we provide theoretical results showing that our method generalizes semantic entropy. Extensive empirical results demonstrate its effectiveness compared to semantic entropy across two recent LLMs (Phi3 and Llama3) and three common text generation tasks: question answering, text summarization, and machine translation. Our code is available at https://github.com/BigML-CS-UCLA/SNNE.