标题： 基于大语言模型的代码生成中无预言的正确性估计 显示英文标题

标题： Estimating Correctness Without Oracles in LLM-Based Code Generation

摘要： 从自然语言规范生成代码是大型语言模型（LLMs）最成功的应用之一。 然而，它们会产生幻觉： LLMs 生成的输出可能在语法上正确，但事实错误。 在没有现有正确实现（即一个 oracle）的情况下，我们能否量化生成程序正确的可能性？ 在本文中，我们提出了一种不一致性的度量方法，称为不一致性，在没有 oracle 的情况下可以高效估计，并提供了错误的下限，即对于该规范的 LLM 生成程序不正确的概率。 我们的实验表明了非凡的有效性。 对于平均代码生成任务，我们的基于不一致性的方法可以在不报告假阳性的情况下自动识别大约三分之二的错误程序。 事实上，基于 oracle 的 LLM 评估可以被基于不一致性的评估可靠地替代。 特别是，我们发现通过 oracle 被视为正确的程序数量对 LLM 进行排序与通过我们的不一致性被视为正确的程序数量对 LLM 进行排序之间存在非常强的一致性。 显示英文摘要

摘要： Generating code from natural language specifications is one of the most successful applications of Large Language Models (LLMs). Yet, they hallucinate: LLMs produce outputs that may be grammatically correct but are factually incorrect. Without an existing, correct implementation (i.e., an oracle), can we quantify how likely the generated program is correct? In this paper, we propose a measure of incorrectness, called incoherence, that can be estimated efficiently in the absence of an oracle and provides a lower bound on the error, i.e., the probability that the LLM-generated program for that specification is incorrect. Our experiments demonstrate an extraordinary effectiveness. For the average code generation task, our incoherence-based methodology can automatically identify about two-thirds of incorrect programs without reports of false positives. In fact, an oracle-based evaluation of LLMs can be reliably replaced by an incoherence-based evaluation. In particular, we find a very strong agreement between the ranking of LLMs by the number of programs deemed correct via an oracle (pass@1) and the ranking of LLMs by the number of programs deemed correct via our incoherence.