标题： PentestJudge：根据操作要求判断代理行为 显示英文标题

标题： PentestJudge: Judging Agent Behavior Against Operational Requirements

摘要： 我们引入了PentestJudge，这是一个用于评估渗透测试代理操作的系统。 PentestJudge是一个大型语言模型（LLM）作为评判者，能够访问工具，使其可以消耗任意的代理状态轨迹和工具调用历史，以确定安全代理的行为是否符合某些操作标准，这些标准在程序上难以评估。 我们制定了评分标准，使用树状结构将特定环境中的渗透测试任务分层压缩为更小、更简单和更易管理的子任务和标准，直到每个叶节点代表PentestJudge可以评估的简单是/否标准。 任务节点被分解为与操作目标、操作安全性和战术相关的不同类别。 LLM作为评判者的评分与人类领域专家进行比较，作为基准参考，使我们能够使用标准二分类指标（如F1分数）比较它们的相对性能。 我们评估了几种前沿和开源模型作为评判代理，表现最好的模型达到了0.83的F1分数。 我们发现那些在工具使用方面表现更好的模型更接近人类专家。 通过按需求类型对F1分数进行分层，我们发现即使总体分数相似的模型在不同类型的问题上也存在困难，这表明某些模型可能在特定的操作标准上是更好的评判者。 我们发现较弱且成本较低的模型可以评判由更强且更昂贵的模型执行的渗透测试轨迹，这表明对于渗透测试任务来说，验证可能比生成更容易。 我们分享了这种方法，以促进未来的研究，理解评判者全面且可扩展地评估基于人工智能的信息安全代理的过程质量的能力，从而使它们能够在敏感的生产环境中被自信地使用。 显示英文摘要

摘要： We introduce PentestJudge, a system for evaluating the operations of penetration testing agents. PentestJudge is a large language model (LLM)-as-judge with access to tools that allow it to consume arbitrary trajectories of agent states and tool call history to determine whether a security agent's actions meet certain operating criteria that would be impractical to evaluate programmatically. We develop rubrics that use a tree structure to hierarchically collapse the penetration testing task for a particular environment into smaller, simpler, and more manageable sub-tasks and criteria until each leaf node represents simple yes-or-no criteria for PentestJudge to evaluate. Task nodes are broken down into different categories related to operational objectives, operational security, and tradecraft. LLM-as-judge scores are compared to human domain experts as a ground-truth reference, allowing us to compare their relative performance with standard binary classification metrics, such as F1 scores. We evaluate several frontier and open-source models acting as judge agents, with the best model reaching an F1 score of 0.83. We find models that are better at tool-use perform more closely to human experts. By stratifying the F1 scores by requirement type, we find even models with similar overall scores struggle with different types of questions, suggesting certain models may be better judges of particular operating criteria. We find that weaker and cheaper models can judge the trajectories of pentests performed by stronger and more expensive models, suggesting verification may be easier than generation for the penetration testing task. We share this methodology to facilitate future research in understanding the ability of judges to holistically and scalably evaluate the process quality of AI-based information security agents so that they may be confidently used in sensitive production environments.