Title: Zero-Shot Statistical Tests for LLM-Generated Text Detection using Finite Sample Concentration Inequalities Show Chinese title

Title: 基于有限样本集中不等式的零样本统计检验用于检测LLM生成文本

Abstract: Verifying the provenance of content is crucial to the function of many organizations, e.g., educational institutions, social media platforms, firms, etc. This problem is becoming increasingly challenging as text generated by Large Language Models (LLMs) becomes almost indistinguishable from human-generated content. In addition, many institutions utilize in-house LLMs and want to ensure that external, non-sanctioned LLMs do not produce content within the institution. In this paper, we answer the following question: Given a piece of text, can we identify whether it was produced by a particular LLM or not? We model LLM-generated text as a sequential stochastic process with complete dependence on history. We then design zero-shot statistical tests to (i) distinguish between text generated by two different known sets of LLMs $A$ (non-sanctioned) and $B$ (in-house), and (ii) identify whether text was generated by a known LLM or generated by any unknown model, e.g., a human or some other language generation process. We prove that the type I and type II errors of our test decrease exponentially with the length of the text. For that, we show that if $B$ generates the text, then except with an exponentially small probability in string length, the log-perplexity of the string under $A$ converges to the average cross-entropy of $B$ and $A$. We then present experiments using LLMs with white-box access to support our theoretical results and empirically examine the robustness of our results to black-box settings and adversarial attacks. In the black-box setting, our method achieves an average TPR of 82.5\% at a fixed FPR of 5\%. Under adversarial perturbations, our minimum TPR is 48.6\% at the same FPR threshold. Both results outperform all non-commercial baselines. See https://github.com/TaraRadvand74/llm-text-detection for code, data, and an online demo of the project. Show Chinese abstract

Abstract: 验证内容的来源对于许多组织的功能至关重要，例如教育机构、社交媒体平台、公司等。 这个问题变得越来越具有挑战性，因为由大型语言模型（LLMs）生成的文本几乎与人工生成的内容无法区分。 此外，许多机构使用内部的LLMs，并希望确保外部未经批准的LLMs不会在机构内生成内容。 在本文中，我们回答以下问题：给定一段文本，我们能否识别它是否是由特定的LLM生成的？ 我们将LLM生成的文本建模为一个完全依赖于历史的序列随机过程。 然后，我们设计零样本统计检验来 (i) 区分由两个不同的已知LLM集合$A$（未经批准的）和$B$（内部的）生成的文本，以及 (ii) 识别文本是否是由已知的LLM生成的，或者是否由任何未知模型生成，例如人类或其他语言生成过程。 我们证明了我们测试的第一类错误和第二类错误随着文本长度的增加呈指数级减少。 为此，我们证明了如果$B$生成文本，则除了在字符串长度上呈指数小概率外，$A$下字符串的对数困惑度收敛到$B$和$A$的平均交叉熵。 然后，我们使用具有白盒访问权限的LLMs进行实验，以支持我们的理论结果，并经验性地检验我们的结果在黑盒设置和对抗攻击下的鲁棒性。 在黑盒设置中，我们的方法在固定FPR为5%的情况下，平均TPR为82.5%。 在对抗扰动下，我们在相同FPR阈值下的最小TPR为48.6%。 这两个结果都优于所有非商业基线。 请查看 https://github.com/TaraRadvand74/llm-text-detection 获取代码、数据和项目的在线演示。