标题： Text-ADBench：基于LLMs嵌入的文本异常检测基准 显示英文标题

标题： Text-ADBench: Text Anomaly Detection Benchmark based on LLMs Embedding

摘要： 文本异常检测是自然语言处理（NLP）中的关键任务，其应用范围涵盖欺诈检测、虚假信息识别、垃圾信息检测和内容审核等。尽管大型语言模型（LLMs）和异常检测算法取得了显著进展，但缺乏对文本数据上现有异常检测方法进行评估的标准且全面的基准，限制了严格比较和创新方法的发展。本工作进行了全面的实证研究，并引入了一个文本异常检测的基准，利用来自多种预训练语言模型的嵌入，在广泛的各种文本数据集上进行实验。我们的工作系统地评估了基于嵌入的文本异常检测效果，包括（1）早期语言模型（GloVe，BERT）；（2）多个LLM（LLaMa-2，LLama-3，Mistral，OpenAI（small，ada，large））；（3）多领域文本数据集（新闻，社交媒体，科技论文）；（4）全面的评估指标（AUROC，AUPRC）。我们的实验揭示了一个关键的实证洞察：嵌入质量显著影响异常检测效果，当使用LLM生成的嵌入时，基于深度学习的方法在性能上并不优于传统的浅层算法（例如，KNN，孤立森林）。此外，我们观察到跨模型性能矩阵具有很强的低秩特性，这使得在实际应用中能够高效地进行快速模型评估（或嵌入评估）和选择。此外，通过开源我们的基准工具包，其中包括不同模型的所有嵌入和代码，地址为https://github.com/jicongfan/Text-Anomaly-Detection-Benchmark，本工作为未来在稳健和可扩展的文本异常检测系统方面的研究提供了基础。 显示英文摘要

摘要： Text anomaly detection is a critical task in natural language processing (NLP), with applications spanning fraud detection, misinformation identification, spam detection and content moderation, etc. Despite significant advances in large language models (LLMs) and anomaly detection algorithms, the absence of standardized and comprehensive benchmarks for evaluating the existing anomaly detection methods on text data limits rigorous comparison and development of innovative approaches. This work performs a comprehensive empirical study and introduces a benchmark for text anomaly detection, leveraging embeddings from diverse pre-trained language models across a wide array of text datasets. Our work systematically evaluates the effectiveness of embedding-based text anomaly detection by incorporating (1) early language models (GloVe, BERT); (2) multiple LLMs (LLaMa-2, LLama-3, Mistral, OpenAI (small, ada, large)); (3) multi-domain text datasets (news, social media, scientific publications); (4) comprehensive evaluation metrics (AUROC, AUPRC). Our experiments reveal a critical empirical insight: embedding quality significantly governs anomaly detection efficacy, and deep learning-based approaches demonstrate no performance advantage over conventional shallow algorithms (e.g., KNN, Isolation Forest) when leveraging LLM-derived embeddings.In addition, we observe strongly low-rank characteristics in cross-model performance matrices, which enables an efficient strategy for rapid model evaluation (or embedding evaluation) and selection in practical applications. Furthermore, by open-sourcing our benchmark toolkit that includes all embeddings from different models and code at https://github.com/jicongfan/Text-Anomaly-Detection-Benchmark, this work provides a foundation for future research in robust and scalable text anomaly detection systems.