Title: CREW-WILDFIRE: Benchmarking Agentic Multi-Agent Collaborations at Scale Show Chinese title

Title: CREW-WILDFIRE：大规模代理多代理协作的基准测试

Abstract: Despite rapid progress in large language model (LLM)-based multi-agent systems, current benchmarks fall short in evaluating their scalability, robustness, and coordination capabilities in complex, dynamic, real-world tasks. Existing environments typically focus on small-scale, fully observable, or low-complexity domains, limiting their utility for developing and assessing next-generation multi-agent Agentic AI frameworks. We introduce CREW-Wildfire, an open-source benchmark designed to close this gap. Built atop the human-AI teaming CREW simulation platform, CREW-Wildfire offers procedurally generated wildfire response scenarios featuring large maps, heterogeneous agents, partial observability, stochastic dynamics, and long-horizon planning objectives. The environment supports both low-level control and high-level natural language interactions through modular Perception and Execution modules. We implement and evaluate several state-of-the-art LLM-based multi-agent Agentic AI frameworks, uncovering significant performance gaps that highlight the unsolved challenges in large-scale coordination, communication, spatial reasoning, and long-horizon planning under uncertainty. By providing more realistic complexity, scalable architecture, and behavioral evaluation metrics, CREW-Wildfire establishes a critical foundation for advancing research in scalable multi-agent Agentic intelligence. All code, environments, data, and baselines will be released to support future research in this emerging domain. Show Chinese abstract

Abstract: 尽管在基于大型语言模型（LLM）的多智能体系统方面取得了快速进展，但当前的基准测试在评估其在复杂、动态、现实任务中的可扩展性、鲁棒性和协调能力方面仍显不足。 现有的环境通常专注于小规模、完全可观测或低复杂度的领域，限制了它们在开发和评估下一代多智能体代理AI框架中的实用性。 我们引入了CREW-Wildfire，一个开源基准，旨在弥补这一差距。 CREW-Wildfire建立在人类-AI团队协作的CREW仿真平台上，提供了程序生成的火灾响应场景，具有大地图、异构智能体、部分可观测性、随机动力学和长时程规划目标。 该环境通过模块化的感知和执行模块支持低级控制和高级自然语言交互。 我们实现了并评估了几种最先进的基于LLM的多智能体代理AI框架，揭示了显著的性能差距，突显了在不确定性下大规模协调、通信、空间推理和长时程规划方面的未解挑战。 通过提供更真实的复杂性、可扩展的架构和行为评估指标，CREW-Wildfire为推进可扩展多智能体代理智能的研究奠定了关键基础。 所有代码、环境、数据和基线都将发布，以支持该新兴领域未来的 research。