Title: BlenderGym: Benchmarking Foundational Model Systems for Graphics Editing Show Chinese title

Title: BlenderGym：用于图形编辑的基础模型系统基准测试

Abstract: 3D graphics editing is crucial in applications like movie production and game design, yet it remains a time-consuming process that demands highly specialized domain expertise. Automating this process is challenging because graphical editing requires performing a variety of tasks, each requiring distinct skill sets. Recently, vision-language models (VLMs) have emerged as a powerful framework for automating the editing process, but their development and evaluation are bottlenecked by the lack of a comprehensive benchmark that requires human-level perception and presents real-world editing complexity. In this work, we present BlenderGym, the first comprehensive VLM system benchmark for 3D graphics editing. BlenderGym evaluates VLM systems through code-based 3D reconstruction tasks. We evaluate closed- and open-source VLM systems and observe that even the state-of-the-art VLM system struggles with tasks relatively easy for human Blender users. Enabled by BlenderGym, we study how inference scaling techniques impact VLM's performance on graphics editing tasks. Notably, our findings reveal that the verifier used to guide the scaling of generation can itself be improved through inference scaling, complementing recent insights on inference scaling of LLM generation in coding and math tasks. We further show that inference compute is not uniformly effective and can be optimized by strategically distributing it between generation and verification. Show Chinese abstract

Abstract: 3D图形编辑在电影制作和游戏设计等应用中至关重要，但仍然是一个耗时的过程，需要高度专业化的领域知识。 自动化这一过程具有挑战性，因为图形编辑需要执行各种任务，每项任务都需要不同的技能组合。 最近，视觉语言模型（VLM）已成为自动化编辑过程的强大框架，但由于缺乏全面的基准测试，这些模型的发展和评估受到限制，该基准测试需要人类级别的感知并呈现现实世界的编辑复杂性。 在这项工作中，我们提出了BlenderGym，这是首个针对3D图形编辑的全面VLM系统基准。 BlenderGym通过基于代码的3D重建任务来评估VLM系统。 我们评估了闭源和开源VLM系统，并观察到即使是最先进的VLM系统在相对容易的任务上也难以与Blender用户匹敌。 借助BlenderGym，我们研究了推理扩展技术如何影响VLM在图形编辑任务中的性能。 值得注意的是，我们的发现表明，用于指导生成扩展的验证器本身也可以通过推理扩展得到改进，这补充了最近关于编码和数学任务中LLM生成推理扩展的见解。 我们进一步展示了推理计算并非始终有效，可以通过在生成和验证之间战略性地分配它来优化。