Title: Towards Robust Agentic CUDA Kernel Benchmarking, Verification, and Optimization Show Chinese title

Title: 面向鲁棒的代理CUDA内核基准测试、验证和优化

Abstract: Recent advances in large language models (LLMs) demonstrate their effectiveness in scaling test-time compute for software engineering tasks. However, these approaches often focus on high-level solutions, with limited attention to optimizing low-level CUDA kernel implementations. Additionally, existing kernel generation benchmarks suffer from exploitable loopholes and insufficient diversity in testing conditions, hindering true generalization assessment. To address these limitations, we introduce robust-kbench, a new benchmark for rigorous evaluation of kernel performance and correctness across varied scenarios. Furthermore, we present a comprehensive agentic framework that automates CUDA kernel discovery, verification, and optimization. This pipeline enables frontier LLMs to translate torch code to CUDA kernels and iteratively improve their runtime within our robust evaluation setting. Our sequential workflow first translates PyTorch code into equivalent CUDA kernels. It then optimizes their runtime using a novel evolutionary meta-generation procedure tailored to the CUDA ecosystem, guided by LLM-based verifiers for correctness and efficient filtering. Evaluated on robust-kbench, our approach produces CUDA kernels outperforming torch implementations for practical applications, including forward and backward passes. It can fuse operations and deploy various runtime optimization strategies. The verifier workflow accurately classifies incorrect kernels, enhancing hardware verification efficiency. Show Chinese abstract

Abstract: 近年来大型语言模型（LLMs）的进展展示了它们在软件工程任务中扩展运行时计算的有效性。 然而，这些方法通常关注高层次解决方案，对优化低层次CUDA内核实现的关注有限。 此外，现有的内核生成基准测试存在可被利用的漏洞，并且测试条件多样性不足，阻碍了真正的泛化评估。 为解决这些限制，我们引入了robust-kbench，这是一个新的基准测试，用于在各种场景下对内核性能和正确性进行严格评估。 此外，我们提出了一种全面的代理框架，可自动完成CUDA内核发现、验证和优化。 该流程使前沿LLMs能够将torch代码转换为CUDA内核，并在我们的鲁棒评估环境中迭代改进其运行时间。 我们的顺序工作流首先将PyTorch代码转换为等效的CUDA内核。 然后，使用一种针对CUDA生态系统的新型进化元生成过程优化其运行时间，由基于LLM的验证器指导以确保正确性和高效过滤。 在robust-kbench上评估，我们的方法生成的CUDA内核在实际应用中优于torch实现，包括前向和后向传递。 它可以融合操作并部署各种运行时优化策略。 验证器工作流能准确分类错误的内核，提高硬件验证效率。