标题： 位置：人工智能竞赛为生成式人工智能评估提供金标准的实证严谨性 显示英文标题

标题： Position: AI Competitions Provide the Gold Standard for Empirical Rigor in GenAI Evaluation

摘要： 在这篇立场文件中，我们观察到生成式人工智能（Generative AI）中的经验评估正处于危机点，因为传统的机器学习评估和基准测试策略不足以满足评估现代生成式人工智能模型和系统的需求。 存在许多原因，包括这些模型通常具有几乎无限的输入和输出空间，通常没有明确的ground truth目标，并且通常表现出基于先前模型输出上下文的强大反馈回路和预测依赖性。 除了这些问题之外，我们认为 {\emleakage } 和 {\em 污染} 的问题是事实上生成式人工智能评估中最重要和最困难的问题。 有趣的是，人工智能竞赛领域已经开发出有效的措施和实践来对抗比赛环境中不良行为者的作弊行为，从而防止信息泄露。 这使得人工智能竞赛成为特别有价值（但未充分利用）的资源。 现在是该领域将人工智能竞赛视为生成式人工智能评估的经验严谨性的黄金标准的时候了，并且应该根据其价值来利用和收获它们的结果。 显示英文摘要

摘要： In this position paper, we observe that empirical evaluation in Generative AI is at a crisis point since traditional ML evaluation and benchmarking strategies are insufficient to meet the needs of evaluating modern GenAI models and systems. There are many reasons for this, including the fact that these models typically have nearly unbounded input and output spaces, typically do not have a well defined ground truth target, and typically exhibit strong feedback loops and prediction dependence based on context of previous model outputs. On top of these critical issues, we argue that the problems of {\em leakage} and {\em contamination} are in fact the most important and difficult issues to address for GenAI evaluations. Interestingly, the field of AI Competitions has developed effective measures and practices to combat leakage for the purpose of counteracting cheating by bad actors within a competition setting. This makes AI Competitions an especially valuable (but underutilized) resource. Now is time for the field to view AI Competitions as the gold standard for empirical rigor in GenAI evaluation, and to harness and harvest their results with according value.