标题： 通过强化学习的代理安全对齐 显示英文标题

标题： Agent Safety Alignment via Reinforcement Learning

摘要： 自主大型语言模型（LLM）代理的出现，能够使用工具，引入了超出传统对话滥用的新安全风险。 这些代理被赋予执行外部功能的能力，容易受到用户发起的威胁（例如，对抗性提示）和工具发起的威胁（例如，受损工具的恶意输出）。 在本文中，我们提出了第一个针对使用工具的代理的统一安全对齐框架，使模型能够通过结构化推理和沙箱强化学习处理这两种威胁渠道。 我们引入了一个三模态分类法，包括良性、恶意和敏感，适用于用户提示和工具响应，并定义了一个策略驱动的决策模型。 我们的框架采用了一个自定义设计的沙箱环境，模拟真实世界的工具执行，并允许细粒度的奖励塑造。 通过在公共和自建基准上的广泛评估，包括Agent SafetyBench、InjecAgent和BFCL，我们证明了我们的安全对齐代理在增强对安全威胁的抵抗力的同时，保持了在良性任务上的强大实用性。 我们的结果表明，安全性和有效性可以共同优化，为自主LLM代理的可信部署奠定了基础。 显示英文摘要

摘要： The emergence of autonomous Large Language Model (LLM) agents capable of tool usage has introduced new safety risks that go beyond traditional conversational misuse. These agents, empowered to execute external functions, are vulnerable to both user-initiated threats (e.g., adversarial prompts) and tool-initiated threats (e.g., malicious outputs from compromised tools). In this paper, we propose the first unified safety-alignment framework for tool-using agents, enabling models to handle both channels of threat via structured reasoning and sandboxed reinforcement learning. We introduce a tri-modal taxonomy, including benign, malicious, and sensitive for both user prompts and tool responses, and define a policy-driven decision model. Our framework employs a custom-designed sandbox environment that simulates real-world tool execution and allows fine-grained reward shaping. Through extensive evaluations on public and self-built benchmarks, including Agent SafetyBench, InjecAgent, and BFCL, we demonstrate that our safety-aligned agents significantly improve resistance to security threats while preserving strong utility on benign tasks. Our results show that safety and effectiveness can be jointly optimized, laying the groundwork for trustworthy deployment of autonomous LLM agents.