Title: Defense Against Prompt Injection Attack by Leveraging Attack Techniques Show Chinese title

Title: 利用攻击技术防御提示注入攻击

Abstract: With the advancement of technology, large language models (LLMs) have achieved remarkable performance across various natural language processing (NLP) tasks, powering LLM-integrated applications like Microsoft Copilot. However, as LLMs continue to evolve, new vulnerabilities, especially prompt injection attacks arise. These attacks trick LLMs into deviating from the original input instructions and executing the attacker's instructions injected in data content, such as retrieved results. Recent attack methods leverage LLMs' instruction-following abilities and their inabilities to distinguish instructions injected in the data content, and achieve a high attack success rate (ASR). When comparing the attack and defense methods, we interestingly find that they share similar design goals, of inducing the model to ignore unwanted instructions and instead to execute wanted instructions. Therefore, we raise an intuitive question: Could these attack techniques be utilized for defensive purposes? In this paper, we invert the intention of prompt injection methods to develop novel defense methods based on previous training-free attack methods, by repeating the attack process but with the original input instruction rather than the injected instruction. Our comprehensive experiments demonstrate that our defense techniques outperform existing training-free defense approaches, achieving state-of-the-art results. Show Chinese abstract

Abstract: 随着技术的进步，大型语言模型（LLMs）在各种自然语言处理（NLP）任务中取得了显著的性能，推动了像Microsoft Copilot这样的LLM集成应用的发展。 然而，随着LLMs的不断发展，新的漏洞出现，特别是提示注入攻击。 这些攻击欺骗LLMs偏离原始输入指令，并执行数据内容中注入的攻击者指令，例如检索结果。 最近的攻击方法利用LLMs的指令遵循能力以及它们无法区分数据内容中注入的指令的能力，实现了高攻击成功率（ASR）。 在比较攻击和防御方法时，我们发现它们具有相似的设计目标，即诱导模型忽略不需要的指令并执行需要的指令。 因此，我们提出一个直观的问题：这些攻击技术能否用于防御目的？ 在本文中，我们反转提示注入方法的意图，通过重复攻击过程但使用原始输入指令而非注入指令，基于之前的无训练攻击方法开发新的防御方法。 我们的全面实验表明，我们的防御技术优于现有的无训练防御方法，达到了最先进的结果。