Title: Towards Inclusive Toxic Content Moderation: Addressing Vulnerabilities to Adversarial Attacks in Toxicity Classifiers Tackling LLM-generated Content Show Chinese title

Title: 面向包容性的有害内容审核：解决有害分类器对对抗攻击的脆弱性，处理大语言模型生成的内容

Abstract: The volume of machine-generated content online has grown dramatically due to the widespread use of Large Language Models (LLMs), leading to new challenges for content moderation systems. Conventional content moderation classifiers, which are usually trained on text produced by humans, suffer from misclassifications due to LLM-generated text deviating from their training data and adversarial attacks that aim to avoid detection. Present-day defence tactics are reactive rather than proactive, since they rely on adversarial training or external detection models to identify attacks. In this work, we aim to identify the vulnerable components of toxicity classifiers that contribute to misclassification, proposing a novel strategy based on mechanistic interpretability techniques. Our study focuses on fine-tuned BERT and RoBERTa classifiers, testing on diverse datasets spanning a variety of minority groups. We use adversarial attacking techniques to identify vulnerable circuits. Finally, we suppress these vulnerable circuits, improving performance against adversarial attacks. We also provide demographic-level insights into these vulnerable circuits, exposing fairness and robustness gaps in model training. We find that models have distinct heads that are either crucial for performance or vulnerable to attack and suppressing the vulnerable heads improves performance on adversarial input. We also find that different heads are responsible for vulnerability across different demographic groups, which can inform more inclusive development of toxicity detection models. Show Chinese abstract

Abstract: 由于大型语言模型（LLMs）的广泛应用，在线生成的内容数量大幅增长，这给内容审核系统带来了新的挑战。 传统的内容审核分类器通常是在人类生成的文本上进行训练的，由于LLM生成的文本偏离了它们的训练数据以及旨在规避检测的对抗攻击，导致出现误分类。 当前的防御策略是被动而非主动的，因为它们依赖于对抗训练或外部检测模型来识别攻击。 在本工作中，我们旨在识别导致误分类的毒性分类器的脆弱组件，并提出一种基于机制可解释性技术的新策略。 我们的研究聚焦于微调的BERT和RoBERTa分类器，在涵盖各种少数群体的多样化数据集上进行测试。 我们使用对抗攻击技术来识别脆弱的电路。 最后，我们抑制这些脆弱的电路，从而提高对对抗攻击的性能。 我们还提供了这些脆弱电路的人口统计层面的见解，揭示了模型训练中的公平性和鲁棒性差距。 我们发现，模型有不同的头部，这些头部对于性能至关重要或容易受到攻击，抑制脆弱的头部可以提高对抗输入的性能。 我们还发现，不同的头部在不同的人口统计群体中负责脆弱性，这可以为毒性检测模型的更包容性开发提供指导。