标题： MERA代码：跨任务代码生成的统一框架 显示英文标题

标题： MERA Code: A Unified Framework for Evaluating Code Generation Across Tasks

摘要： 在大型语言模型方面的进展提高了软件工程中的任务自动化；然而，当前的评估主要集中在自然语言任务上，忽视了代码质量。 大多数基准测试更注重高层次的推理而非可执行代码和现实世界的表现，这导致了对这些模型在生产环境中真正能力和风险的理解存在空白。 为了解决这个问题，我们提出了MERA Code，这是MERA基准家族的新成员，专门用于评估最新代码生成大型语言模型在俄语中的表现。 该基准包括11个评估任务，涵盖8种编程语言。 我们提出的评估方法具有一个分类法，概述了模型完成这些任务所需的实际编码技能。 该基准包括一个开源代码库，供用户进行MERA评估，一个与各种编程环境兼容的评分系统，以及一个具有排行榜和提交系统的平台。 我们评估了开放大型语言模型和前沿API模型，分析了它们在非英语语言中的实际编码任务上的局限性。 我们公开发布MERA，以指导未来的研究，预测模型开发中的突破性功能，并标准化评估流程。 显示英文摘要

摘要： Advancements in LLMs have enhanced task automation in software engineering; however, current evaluations primarily focus on natural language tasks, overlooking code quality. Most benchmarks prioritize high-level reasoning over executable code and real-world performance, leaving gaps in understanding true capabilities and risks associated with these models in production. To address this issue, we propose MERA Code, a new addition to the MERA benchmark family, specifically focused on evaluating code for the latest code generation LLMs in Russian. This benchmark includes 11 evaluation tasks that span 8 programming languages. Our proposed evaluation methodology features a taxonomy that outlines the practical coding skills necessary for models to complete these tasks. The benchmark comprises an open-source codebase for users to conduct MERA assessments, a scoring system compatible with various programming environments, and a platform featuring a leaderboard and submission system. We evaluate open LLMs and frontier API models, analyzing their limitations in terms of practical coding tasks in non-English languages. We are publicly releasing MERA to guide future research, anticipate groundbreaking features in model development, and standardize evaluation procedures.